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ABSTRACT 

This  two-volume  document  describes  a  study  which 
focused  On  characterizing  the  elements  involved  in  information 
seeking  and  retrieving,  particularly  in  relation  to  the  cognitive 
decisions  and  human  interactions  inyoiyed  in  online  information 
retrieval.  The  study  Objectives  were  to  conduct  experiments  and 
observations^  under  conditions  as  close  to  real  life  as  possiblei 
related  to:  11)  the  user  context  of  questions  in  information 
retrieval;   (2)  the  structure  and  classification  of  guestions;   (3)  the 
cognitive  traits  aiid  decision  making  of  searchers^  and  (4)  the 
comparative  nature  Of  the  search  of  the  same  question  by  different 
searchers.  Study  results  indicate  that  fluency  in  associations  in 
English  and  in  English  idiOms  seems  to  be  an  important  characteristic 
of  more  effective  searches;  that  items  retrieved  in  searches  in  which 
the  searcher  displayed  results  more  frequently  are  more  likely  to  be 
relevant^  and  that  higher  recall  is  achieved  when  outputs  from 
several  searchers  are  merged.^  Volume  1  describes  the  study  background 
and  presents  the  study  methodology;  basic  background  data;^  a  summary 
of  the  results;  and  statistical  analyses  of  study  data  organized 
according  to  the  five  Variables  of  information  seeker  or  user, 
question,  searcher,  search,  and  retrieved  items.  An  executive 
summary,  2  figures,  and  60  tables  are  included,  as  well  as  a  48-item 
bibliography.  Volume  2  comprises  nine  appendices,  including 
questionnaire  statements  provided  by  the  users,  raw  retrieval  results 
for  each  searcher  and  question,  forms  used  in  the  study  process,  and 
study  procedures  and  flowcharts.  (KM) 
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EXECUTIVE  SUMMARY 
Final  Report  on  NSF  Grant  iST  85-05411 

EXPERIMENTS  ON  THE  GOGNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 
Tefko  Saracevic,  Paul  Kantor,  Alice  Y.  Chamis,  Donna  Trivison 


 The  aim  of  the  study  was  to  contribute  to  the  formal,  scientific 

characterization  of  the  elements  involved  in  information  seeking  and 
retrieving,  particularly  in  relation  to  the  cognitive  decisions  and 
human  interactions  involved.    The  objectives  were  to  conduct 
experiments  and  observations  under  as  real-life  conditions  as 
possible  related  to:   (1)  user  context  of  questions  in  information 
retrieval;   (2)  the  structure  and  classification  of  £uestion£;  (3) 
cognitive  traits  and  decisibh-making  of  searchers;  and  (4)  different 
searches  of  the  same  question.    Models  and  measures  were  developed  to 
reflect  the  problein-solvihg  approaches  of  users  and  searchers.  In 
that,  the  study  relates  to  problem  oriented  research  in  information 
science,  cognitive  science^  and  artificial  intelligence. 

The  fbllbwing  aspects  of  information  seeking  and  information 
searching  were  studied  as  grouped  in  five  general  classes  of  the 
entities  involved : 

1.  User;  effects  of  the  context  of  questions  and  constraints 
placed  on  questions 

2.  Question:  structure  and  classification  assignments  by 
different  judges  and  the  effect  of  various  classes 

3.  Searcher;  effects  of  cognitive  traits  and  frequency  of  online 
experience 

4.  Search;  effects  of  different  types  of  searches^  overlap 
between  searches  of  the  same  question  in  selection  of  search 
terms  and  items  retrieved;  efficiency  and  effectiveness  of 
searches 

5.  Xt^g— re t^rl^ve^ ;  magnitude  of  retrieval  of  relevant  and 
nonrelevant  items ;  effects  of  other  variables  on  the  chances 
retrieved  items  were  relevant 

Forty  users  each  posed  bhe  written  question  related  to  their 
brigbing  research  br  work.     In  additibh^  during  an  interview  each  user 
supplied  a  tape  recotcied  statement  on  the  underlying  problem  of 
his/her  research.     They  also  marked  measures  oh  the  context  of  the 
question  dealing  with  problem  definition,  intent^  internal,  knowledge, 
and_ public  knbwledge  estimate.     Forty  searchers  were  assembled:  36  so 
called  'outside*  searchers  who  searched  five  or  six  questions  based 
on  the  user's  written  question,  three  project  or  staff  searchers  who 
conducted  four  different  types  of  searches  ('project  searches')  and 
classified  questions  and  one  additional  judge  for  question 

v 


classification i  The  project  searches  were  based  dii:  1.  the  user's 
tape  recorded  problem  statement  only       2,  taped  prdbiem  statement 

the  written  question  3.  terms  from  the  written  quest.'bn  only 
without  elaboration >  and  4.  terms  from  the  written  question  plus 
elaboration  by  thesaurus.    Each  searcher  was  tested  on  three 
cognitive  tests:   (1)  Remote  AssdciatesTest  (RAT)  designed  to  test 
ability  in  making  word  asspciatiphsi   (2)  One  test  from  a  group  of 
test  called  collectively  the  Employee  Aptitude  Survey  (EAS)^  the 
Symbolic  Reasoning  Test  designed  to  test  the  ability  to  make 
deductive  inferences  from  symbolic  inequalities;   (3)  Learning  Style 
Inventory  (LSI)  designed  to  characterize  an  individual  according  to 
preferred  style  of  learning.    Searchers  also  indicated  a  frequency  of 
DIALOG  use.     Searching  was  done  on  DIALOG:  a  single  DIALOG  database 
was  searched  for  each  question. 

Each  of  the  40  questions  was  searched  by  five  outside  searchers 
and  four  project  searches i    The  output  from  these  nine  searches  was 
merged  into  a  union ^  l^ei,  duplicates  were  eliiainatedi    The  union  was 
then  sent  to  the  user  for  evaluations     If  the  union  exceeded  150 
retrieved  items,  only  the  first  (most  recent)  150  items  were  sent  to 
the  users    This  was  done  in  order  to  avoid  user  overload  and  to 
ensure  return.    Users  indicated  whether  each  item  was  relevant, 
paritally  relevant,  or  nonrelevant.    Additionally,  users  scored  five 
atility  measures :  time  spent  in  evaluation,  dollar  value,  worth  of 

problem  resolutiorij  and  overall 
satisfaction.    These  evaluations  were  used  as  benchmarks  for  figuring 
precision  and  recall  of  the  searches  and  for  study  of  the  other 
variables.    All  together  90  variables  were  defined. 

For  each  question  there  were  nine  searches, (five  outside 
searches  and  four  project  searches)^  resulting  in  a  grand  total  of 
360  searches  (200  outside  searches  and  160  project  searches)  for  the 
40  questions.    The  sim  of  items  retrieved  (cbuhtirig  search  by  search 
for  360  searches)  before  duplicates  were  eliminated  included  8956 
evaluated  i terns ^  and  after  duplicates  were  elimin ted,  the  total 
number  of  unique  items  was  5411.    Of  the  5411  unique  items,  1343  or 
25%  were  judged  relevant  by _ the  users,  1448  or  27%  were  judged 
partially  relevant,  and  2620  or  41%  were  judged  nonrelevant. 

Statistical  analyses  in  search  of  variables  which  could  provide 
explanations  of  the  observed  search  processes  were  done  on  two 
levels:  search-wise  and  item-wise.    On  the  search-wise  level,  impact 
on  precision  and  recall  was  considered^    On  the  item-wise  level, 
impact  of  variables  on  the  chances  that  retrieved  items  were  relevant 
as  opposed  to  nonrelevant  was  considered.    For  all  except  one 
analysis,  relevant  (R)  and  partially  relevant  (pR)  items  were 
considered  together  (i.e.,  R  +  pR).    A  sdmmary  of  results  follows. 
Wherever  means  are  given,  they  should  be  interpreted  with  caution 
because  na  distribution  was  normal  (bell  shaped):  all  of  the 
distributions  were  either  skewed  in  one  direction  or  had  several 
peaks • 

i .    Users  *  indicatloth^f  question  context .    The  context 
elements:   (1)  problem  definition  (how  clearly  defined),  (2)  intent 

vi 
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(how  specific  the  intended  use),  and  (3)  estimate  of  public  knowledge 
(probability  that  public  infpmatibh  exists)  had  a  significant  impact 
on  the  chance  of  retrieval  of  relevant  items.    When  a  problem  was 
considered  well  defined,  a  retrieved  item  was  17%  more  likely  to  be 
relevant J  when  intent  was  'open  to  many  avenues'  there  is  a  27% 
decline  in  the  chance _ a  retrieved  item  was  relevant;  and  when  there 
was  a  high  estimate  of  the  existence  of  public  knowledge^  the  chance?; 
for  relevance  increased  110%.    High  or  low  internal  knowledge 
(estimate  by  the  user  of  the  extent  of  personal  knowledge)^  the 
fourth  user  context  measure^  had  no  significant  impact  on  relevance. 

2.  Users'  cdhstratnts  on  questions i    Specifications  about 
language  and  years  of  publication  had  a  significant  impact  on  the 
chances  that  a  retrieved  item  was  relevant,  while  type  of  application 
and  request  for  a  precise  or  broad  search  did  not.    None  of  the 
variables  had  a  significant  impact  on  the  precision  of  the  union  of 
retrieved  items  submitted  to  the  user.     (Comparative  recall  for  given 
searches  can  be  calculated,  but  recall  for  the  question  as  a  whole, 
^^?_'>_^9?  9^^P^^.»  cannot  be  calculated  because  we  do  hot 
know  what  relevant  items  were  left  in  the  DIALOG  file).    If  the 
language  was  not  restricted  to  English^  chances  for  relevance  were 
reduced  by  37%  and  enhanced  by  28%  if  there  were  no  restrictions  on 
year  of  publication.    The  mean  precisions  for  various  constraints 
were:  by  application:  50%  for  faculty  research,  49%_ graduate  study, 
49%  industry  and  66%  other;  by  language:  English  only  56%,  any 
language  42%;  by  years:  last  five  years  58%,  no  limit  48%. 

3.  Consistency  and  estimate  of  user  context  by  searchers. 
Searcher  judgements  of  question  context  was  compared  with  user 
judgements.    There  was.  substantial  agreement  on  how  well  the  problem 
was  defined,  followed  by  agreement  oh  the  specificity  of  the  intended 
use.    Agreement  on  estimates  of  the  existence  of  public  knowledge 
about  the  problem  was  lb\ir-^sers  judged  it  considerably  higher i  User 
estimates.  pf_  their  internal  knowledge  about  the  problem  was  as 
expected  higher  than  searcher  estlamteSi    Thus,  to  some  degree, 
searchers  are  able  to  estimate  user  assessments  of  context.  However, 
only  intent  and  public  knowledge  had  an  effect  on  chances  for 
relevance.    When  searchers  assessed  user  intent  as  narro\d.y  defined, 
ah  item  retrieved  was  32%  more  likely  to  be  relevant i    When  searchers 
assessed  substantial  public  knowledge,  chances  for  relevance  improved 
a  small  11%. 


4.    Consistency  of  judgement  on  question  elassif  icatdon^  between 
two  judges i    When  given  the  same  question  to  classify  using  a  scheme 
developed  for  the  study  there  was  substantial  agreement  on  a  number 
of  categories:  (1)  general  subject  domain  of  the  question  (number  of 
DIAtiNDEX  categories);  (2)  clarity  of  semantics  and  syntax;  (3) 
apecificity  of  the  subject  of  the  questions,  and  (4)  constraints. 
There  was  poor  agreement  on:  (1)  specificity  of  the  query  about  the 
subject  of  the  question,  and  (2)  complexity,  or,  the  number  of 
concepts  involved  in  the  question.    Overall,  qiiestidh  classification 
as  specified  seems  to  be  valid,  however,  more  cdhslstency  tests  are 
needed. 
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5*  >  Impact— o£-^Qestlon-cla88J.ficat4Qn-categorleg^>    There  was 
some  significant  impact  on  retrieval  of  relevant  items  by  queistions 
that  were  judged  high  on  clarity  and  high  on  the  number  of 
constraints  and  concepts*     In  other  words,  clearly  structured 
questions,  with  a  greater  xitamber  of  concepts  and  constraints 
significantly  favor  the  chances  of  finding  relevant  items.  Other 
categorizations  showed  little  or  no  impact. 

6.  Searcher  characteristics.     Some  cognitive  traits  showed 
positive  impact  and  some  negative  impact  oh_ relevance  of  retrieved 
items.    Ah  item  retrieved  by  a_  searcher  with  higher  scores  on  the 
Remote  AssbciatesTest  (English  word  associations^  some  of  which  were 
idiomatic)  was  55%  more  likely  to  be  relevant.     Items  were  11%  less 
likely  to  be  relevant  if  retrieved  by  a  searcher  scoring  high  on  the 
Employee  Aptitude  Survey  (deductive  inference  from  symbolically 
expressed  inequalities).    Searchers  categorized  as  having  a  mode  of 
learning  called  Abstract  ebnceptuallzatlon  (tends  to  learn  by 
abstract  thinking)  had  25%  improved  chances  for  relevance,  wHiie 
searchers  categorized  as  preferring  a  Concrete  Experience  style 
(tends  to  learn  by  doing)  had  28%^decreased  chances  for  relevance. 
95_99?^^?^?^9?_?99^®?_^^__^l^®_i®^5^^^S_ style  test ,  searchers  who 
emphasize  abstractness  over  concreteness  had  28%  improved  chances  for 
relevance i    Therefore,  searchers  with  higher  language  abilities 
(regardless  of  specific  subject)  and/or  preferring  an  abstract  style 
of  learning  have  increased  chances  for  relevance  while  searchers  with 
higher  mathematical--logical  abilities  and/or  a  concrete  experience 
style  of  learning  have  decreased  chances  for  relevance.  Searchers 
who  participated  did  searching  as  part  of  their  professional 
functions,  some  more  and  some  less  frequently,  however,  frequency  of 
DIALOG  searching  did  not  show  a  significant  Impact  on  chances  for 
relevance • 

7.  Search  efficiency  characteristics.    The  most  significant 
positive  Impact  was  observed  in  searches  using  more  cycles.     (A  cycle 
is  defined  as  a_ sequence  of  commands  from  selecting,  terns  to 
displaying  results^  thus  cycling  involves  the  possibility  of  review 
and  feedback.).  Items  retrieved,  by  searches  using  a  greater  number  of 
cycles  were  25%_mbre  likely  to  be  relevant  than  items  from  searches 
using  fewer  cycles.     Significant  negative  impact  was  produced  by: 
greater  amounts  of  preparation  time  (items  were  13%  less  likely  to  be 
relevant) ,  greater  total  time  (36%  less  likely)  and  more  search  terms 
(39%  less  likely).    Thus,  searchers  that  use  more  cycles,  not  overly 
much  preparation  and  total  time  (including  online)  and  not  a  great 
number  of  search  terms  have  a  better  chance  of  producing  relevant 
items.    Otx  the  average  per  search  there  were:  15  commands^  3  cycles^ 
and  10  search  terms  used  after  10  minutes  of  preparation  time,  12 
minutes  online  time,  and  22  minutes  total  time.    Panges  varied  widely 
however^    None  ?f  these  search  characteristics  expl^         Q^^Ff  5% 
of  precison  or  recall  for  the  search  overall  (the  search-wise  level 
of  analysis )i    this  is  an  example  of  the  inability  of  a  search-wise 
level  of  analysis  to  detect  differences. 

  8.    Types  of  searches.    There  was  significant  impact  on  recall 

and  no  impact  on  precision  observed  in  the  four  different  types  of 
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searches  (lie;,  the  project  searches).    Searches  based  dri:  (1)  taped 
problem  statement  by  user  had  32%_meah  recall  and  63%  mea^  precision; 
(ii)  written  question  statement  plus  elaboration  by  thesaurus  had  25% 
mean  recall  and  61%  mean  precision;  (iii)  taped. problem  statement 
p3Jia  written  question  had  23%  mean  recall  and  63%  mean  precision;  and 
(iv)  terms  from  the  writtin  question  with  no  elaboration  had  18%  mean 
recall  and  57%  mean  precision.    The  best  performance  was  on  searches 
done  on  the  basis  of  a  user  explanation  and  discussion  of  his  or  her 
problem  (taped  problem  statement)  and  worst  when  words  from  the 
written  questions  were  usedas  terms  without  elaboration,  as  if  they 
had  been  picked  automatically. 

9.    Overlap  of  search  terms  and  items  retrieved ^     [Note:  As 
recalledj  each  question  was  searched  nine  times:  five  'outside 
searches*  each  done  by  a  different  searcher  and  four  'project 
searches*  based  bndifferent  sources  for  the  search  (see  preceeding 
point)  done  by  staff  searchers.    Overlap  was  studied  only  for  the 
five  'outside*  searches,  i.e.^  comparisons  were  made  of  the  degree  of 
agreement  in  selection  of  search  terms  and  in  items  retrieved  for 
each  pair  of  five  searches  done  for  a  question,    the  number  of  pairs 
compared  was  800  =  5  searches  per  question  X  4  comparisons  for  each 
search  of  that  question  (the  search  was  not  compared  with  itself)  X 
40  questions.] 

There  was  a  surprisingly  low  degree  of  agreement  on  selection  of 
search  terms  between  different  searchers  searching  the  same  question. 
The  mean  agreement  on  search  terms  was  2"^%,  but  the  distribution  was 
skewed  toward  the  low  endi    in  20%  of  the  comparisons,  agreement  was 
10%  or  less  and  in  44%  of  the  comparisons  agreement  was  20%  or  less. 

The  overlap  in  retrieved  items  for  the  same  question  by 
different  searches  was  even  lower  than  the  overlap  observed  in 
selection  of  search  terms:  the  mean  was  16%;  in  59%  of  the 
comparisons  overlap  was  less  than  5%  and  in  66%  of  the  comparisons 
overlap  was  less  than  10%.     In  retrieval  of  relevant  items,  in  59%  of 
the  comparisons  overlap  was  less  than  5%  and  in  64%  of  the 
comparisons  overlap  was  less  than  10%.    Surprisingly,  the  substantial 
disagreement  in  items  retrieved  could  not  be  explained  by 
disagreement  in  search  terms  used.    Only  2.5%  of  the  variation  in 
overlap  of  retrieved  items  could  be  attributed  to  overlap  in  search 
terms.    These  are  some  of  the  most  surprising  findings  of  the  study. 

 1Q»    Odds-^of-xelevance  Itv  retrleval^^  <iuplicat    items.  [Note: 

a  question  was  searched  by  five  different  searchers.     In  this 
analysis,  we  studied  duplicate  retrievals^  i.e.,  the  same  item 
retrieved  by  two,  three,  four,  or  five  searches.     In  all  previous  and 
subsequest  analysis  relevant  (R)  and  partially  relevant  (pR)  items 
were  considered  together  (i.e.,  R  +  pR).    For  this  analysis,  only  the 
number  of  relevant  items  is  being  considered,  excluding  partially 
relevant  items.    This  was  done  to  sharpen  insight  into  the  odds  for 
relevance.] 

The  more  often  an  item  was  retrieved  by  different  searches  for 
the  same  question,  the  more  likely  it  was  to  be  relevant.    For  all 
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retrlevalis  (thoise  retriieved  oiice  arid  those  retieved  more  thari  price) 
the  odds  an  item  was  relevarit  as  opposed  to  ribrirelevarit  were  about 
five  to  ten.     If  ari  item  was  retrieved  only  once  (but  of  a  possible 
five  searches)  the  cbrrespbndirig  bdds  were  four  to  terij  items 
retrieved  twice  had  an  even. chance  of  being  relevarit  or  rionrelevarit; 
for  items  retrieved  three  times  or  more  (i.e.,  three,  fbur^  or  fivie 
times)  the  odds  were  quite  favorable  for  relevance^  16  to  10.  In 
other  words ^  wheri  an  item  is  retrieved  three  or  more  times,  the  odds 
for  relevance  are  much  greater  than  the  odds  for  nonrelevance*  This 
we  consider  the  most  significant  finding  of  the  study* 

11.  Recall  and  precision^    The  mean  recall  and  precision  values 
for  searches  confirm  similar  figures  from  other  studies.    For  360 
searches^  the  mean  recall  was  22%  and  mean  precision  was  57%. 
However^  the  next  finding  is  contrary  to  many  studies  and  contrary  to 
the  eleverden  taw:  when  recall  was  plotted  ^gainst  precision  the  two 
were  not  inversely  related.    To  the  contrary,  when  either  recall  or 
precision  was  considered  as  the  independent  variable,  the  other  rose 
mildly.    As  precision  rose  so  did  recall  (although  not  as  much)  or 
vice  versa,  as  recall  rose  so  did  precision,  also  rather  mildly.  The 
plot  of  recall  and  precision  shows  a  large  amount  of  scatter.  The 
most  important  explanatory  variable  for  precision  was  the  user 
estimate  of  existing    public  knowledge — it  explained  10%  of  the 
variance  observed.    The  next  variable,  which  explained  5%  of  the 
variance,  was  a  searcher  characteristic  measured  by  the  Rembte 
Associates  Test  of  word  association  ability.    No  other  variable  used 
passed  the  test  for  significance.    For  recall,  only  brie  variable  was 
mildly  sigriificarit:  a  cbmbiried  score  bri  the  Learning  Style  Iriveritory 
which  iridicated  ari  iridiyidual*s  emphasis  bf  abstractriess  over 
cbricreteriess  as  a  learriirig  style.    This  variable  explalried  somewhat 
less  thari  5%  of  the  observed  varlatlbn.    A  low_ perceritage  bf 
varlatlbri  Iri  recall  arid  preclslbri  could  be  explalried  by  variables 
used  iri  this  study.    For  the  most  part^  we  still  dp  not  kriow  what 
variables  have  a  large  cbritributbry  effect  on  recall  and  precision. 

12.  Utility  assessmerit  by  users.    Four  but  bf  the  five  utility 
measures  used  in  the  study  correlated  positively  with  ch^ihces  that  a 
retrieved  items  was  relevant  while  the  fifth  measure  had  no  effect. 
Items  in  searches  where  the  user  considered  the  value  of  the  search 
results  above  $75  were  28%  more  likely  to  be  relevant  than  items 
retrieved  in  searches  valued  below  that  amount.    Where  users 
indicated  that  the  results  were  worth  the  same  or  more  than  the  time 
they  spent ^  the  chances  for  relevance  were  104%  greater^    Where  users 
indicated  a  greater  contribution  made  toward  resolution  of  their 
problem^  chances  for  relevance  were  84%  greater.    Where  users 
indicated  a  higher  degree  of  satisfaction^  chances  for  relevance  were 

8?®^???1__^5_^1^?^  ^^^^^•.^^^^'^^^^^.^^.items  submitted  and  user 
utility  scores  seem  to  he  parallel ,  indicating  the  possibility  that 
relevance  and  utility  as  measures  are  not  very  different  from  each 
others 

13.  !&go-level8  of  statistical  analysis .    Of  the  two  levels  of 
analysis  used,  the  item-wise  analysis  (based  on  relevance  of  items 
retrieved)  showed  more  power  for  explanation  and  insight  than  the 
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search-wise  analysis  (based  on  precision  and  recall).    While  the 
item-wise  approach  is  conunon  in  bioiiiedicine>  this  is  the  first  time, 
to  oar  knowledge,  that  it  was  applied  in  information  science. 
Results  recommend  its  use* 

 This  study  addressed  the  questibiL  of  what  is  happening  in 

information  seeking  and  retrieving.  The  results  beg  further  studies 
for  verification,  explanation,  and  generalization  based  on  confirmed 
experimental  evidence. 


Implications  of  this  study  are  many.    They  extend  to  future 
research,  education  and  training  of  searchers,  professional  practice, 
guidance  of  users,  and  most  importantly,  to  design  and  improvement  of 
human-system  interfaces.    To  be  successful^  design  of  future 
intelligent, human-system  interfaces  and  intelligent  information 
systems  will  heed  to  take  into  account  results  from  studies  such  as 
this  one  because  the  human-system  interface  must  first  be  based  on 
the  hxaman  part  of  the  equation  and  then  on  systems  and  technology. 
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PREFACE 


The  aim  of  this  report  is  to  present  in  some  detail  the  data  and 
results  of  a  study  investigating  a  variety  of  aspects  related  to 
questions I  searchers ^  and  searching,  in  the  context  of  information 
retrieval  systeiAs.    The  study  is  a  part  of  a  larger  effort  whose 
collective  aim  is  to  contribute  to  a  formal  or  scientific 
characterization  of  the  elements  involved  in  information  seeking  and 
retrieving,  particularly  in  relation  to  cognitive  decisions  and 
himian-system  interactions. 

This  larger  effort  is  divided  into  three  parts  or  phases.  The 
first  phase  (NSFgraat  1ST  80-15335    conducted  from  1981-83)  was  a 
methodological  investigation  devoted  to  development  of  models, 
methods^  and  measures  suitable  for  collection  of  data  and  observation 
of  a  series  of  variables  involved  in  information  seeking  and 
searching i    The  second  phase  involved  the  study  presented  in  this 
report  (NSF  grant  1ST  85--054 11  and  a  1985-86  DIALOG  grant  for  search 
time,  conducted  1985-86).    This  phase  was  devoted  to  testing  of  the 
aforementioned  models,  methods,  and  measures  and  a  series  of 
quantifiable  observations  made  under  as  real-life  conditions  as_ 
possible.    The  third  phase  (planned  for_ study  in  1987-88)  will  be 
devoted  to  an  exhaustive  analysis  of  collected  data  and  search 
protocols  in  order  to  describe  the  nature  and  objective  patterns  (if 
any)  in  searching  atid  iti  the  relation  between  questions  and  answers 
as  evaluated  by  users  in  order  to  make  recommendations  for  the  design 
of  advanced  human— information  system  interactions  and  for  increased 
system  intelligence. 

As  such,  this  study  and  the  larger  effort  responds  to  an  urgent 
need  for  more  research  and  more  scientific  evidence  related  to 
information  seeking  and  searching.    Online  searching  is  an  ever 
growing  activity.    New  information  systems,  such  as  expert  systems, 
hew  technologies  and  interfaces  will  increase  the  realm  of  users, 
searchers  and  uses •    Unf brttinately  all  this  growth  in  professional 
practice  and  the  widening  of  technological  applications  has  not  been 
accompanied  by  a  commensurate  effort  in  research i    in  this  area, 
science  is  lagging  far  behind  practice  mdtechnolo         While  there 
are  a  number  of  related  research  studies  (enumerated  in  the 
bibliography)  concentrated  in  information  science  and  artificial 
intelligence^  the  whole  area  of  scientific  study  of  human  information 
seeking  and  searching  of  questions  and  question-answering  and  the 
human-system  interface  is  barely  beyond  infancy.    We  are  all  asking 
the  same  question:  What  evidence  do  we  have  and  can  we  collect  that 
will  give  us  some  scientifically  supported  insight  into  what  is 
really  going  on  in  these  processes? 

The  present  study  has  produced  a  wealth  of  data  that  in  itself 
is  a  considerable  and  even  unique  research  resource.    Our  idea  is  to 
expiitoit  further  this  data  and  provide  open  access  to  this  data  to  all 
who  desire  to  use  it  further  (and^  of  course^  have  resources  to  do 
so).    To  this  end  we  have  prepared  this  final  report  in  two  parts. 
The  first  part  contains  descriptions  of  methods,  results^  and 
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conclusions.    The  second  part  Is  composed  of  a  set  of  appendices 
containing  as  much  of  the  *raw/  data  as  we  cotild  reasonably 
reproduce.    The  rest  of  the  data  in  machine-readable  form  is 
deposited  with  the  complete_prbject__archive  reproduced  at  two 
universities rj^ase  Western  Reserve  University  (contact  Paul  Kantdr) 
and  Rutgers^  The  State  University  of  New  Jersey  (contact  Tefkd 
Saracevic)_.    The  first  part^  containing  results^  can  be  read  without 
the  appendices.     In  other  words ^  the  first  part  stands  on  its  own  and 
the  appendices  are  an  invitation  for  verification,  replication  and 
further  in-depth  studies. 

To  our  knowledge,  this  is  the  largest  project  of  its  kind 
anywhere ,  using  the  largest  number  of  real  users  (40)  and  real 
questions  (40),  real  professional  searchers  (40^    real  operating 
databases  and  a  real  operating  information  system  (DIALOG),  the 
number  of: items  retrieved  and  evaluted  by  the  users  for  relevance  to 
their  question  (5411)  is  also  among  the  largest  (if  not  the  largest) 
of  any  similar  study,  including  tests  of  retrieval  systems. 

Stilly  we  cannot  claim  generalizations  beyond  our  sample,  any 
more  than  any  other  similar  study  has  been  able  to  claim.    We  have 
not  done  a  random  selection  of  searchers,  users  and  questions  in  the 
United  States  or  whatever  universe.    However,  we  can  offer  our 
observations  with  the  hope  that  other  studies,  particularly 
replications,  may  confirm,  refute  and/or  enlarge  ujpon  our 
observations . 


Organisgation  of  the  Report 


As  mentioned^  the  report  is  organized  ihtwd  volumes.  This 
volume  contains  the  text|  while  the  second  volume  contains  the 
Appendices  including:  written  question  statements  provided  by  the 
users,   *raw*  retrieval  results  for  each  searcher  and  question,  forms 
used,  and  the  procedures  and  flowcharts  of  the  study. 

The  discussion  is  organized  around  the  five  entities  or  classes 
of  variables  of  interest  in  this  study: 

1 •  Information  seeker  or  user 

2.  Question 

3i  Searcher 

4.  Search 

5.  Retrieved  items 

^^^^.^.P^^'^i^®?         background  for  the  study:  an  introduction 
with  a  definition  of  the  above  entities;  a  summary  of  the  study's 
aim,  objectives,  and  approach;  and  a  review  of  the  literature. 
Each  successive  part  is  organized  along  the  five  entities  of  the 
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study. 


Part  II  Is  devoted  to  methodology ^    The  description  of  the 
models,  measures  and  procedures  used  is  given  in  the  order  of  the 
entities  as  listed  above* 

contains  the  basic  data  giving  background  data  on  iisers 
and  searchers,  and  a  summary  of  results  pertaining  to  efficiency  and 
structure  of  searches,  number  of  items  retrieved  and  their  evaluation 
and  the  resulting  effectiveness  of  searches, 

 Part  iV  is  devoted  to  statistical  analysis.     It  contains  a 

chapter  describing  the  methods  used  for  analysis  and  three  chapters 
on  results:  one  on  consistency  of  judges  in  assigning  question 
classification,  and  assessing  context  of  questions,  another  on 
overlap  in  search  terms  and  results  among  searchers^  and  the  longest 
one  rehearsing  the  correlation  of  variables  among  the  various 
entities  in  terras  of  how  they  affect  retrieval  of  items  Judged 
relevant  by  users. 

Part  V  contains  conclusions  arid  a  discussion  of  results  with 
implications  for  practice  and  research. 
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PART  1.  BACKGROUND 


1.  INTRODUCTION:  PROBLEM  AND  SIGNIFICANCE 


 While  there  is  great  variety  in  the  design  and  operation  of 

retrievai  systems  and  a  great  many  variables  interact  to  affect 
system  performance,  the  set  of  variables  that  deal  with:  (i)  the 
nature  of  questions,  (ii)  the  context  of  informatlbh  seeking^  and 
(^^^5^^?  ?^^^^*^^      ^^^^'c^ingj.  can  and  do  affect  performance  to  the 
extent  that  they  spell  the  difference  between  failure  and  success 
regardless  of  the  design  of  the  system. 

The  problems  associated  with  processes  and  variables  of 
information  seeking  and  searching  are  the  most  complex  of  ail  the 
problems  involved  with  retrieval  systems,  because  of  an  overriding 
influence  of  the  cognitive  aspects.    The  cognitive  dimension 
predominates  in  interactions  between  humans  and  information  systems. 
Yet  and  evenmpre  impbrtahtly,  it  Is  the  least  researched  of  ail  the 
dimensions  of  iriformatlbn  retrieval  systems  of  ail  types i  the 
context  of  requests  involves  a  host  of  variables  in  informaion 
seeking  and  the  handling  of  requests  involves  a  host  of  variables  in 
information  retrieving.    Fidel  and  Sbergei  [l-i]  provided  a  list  of 
over  200  such  variables  grouped  in  eight  broad  classes. 

Of  interest  to  this  study  are  five  general  classes  of  variables 
or  entities  that  deal  with  the: 

!•  Information  seeker  or^jusar      a  person  who:  (1)  is  cdnfrbrited 
with  a  problem  on  the  basis  of  which  a  question  io 
fbrmuiated,,  (ii)  provides  a  context  for  the  question;  and 
(iii)  produces  the  ultimate  evaluation  of  the  retrieval 
results. 

2i  Question  -  a  spacific  text  of  a  request  by  the  user  for 
information  submitted  to  an  informatibii  system. 

3.  Searchey     a  person  performing  the  necessary  question 
analysis,  formulating  the  search  strategy  and  conducting  the 

User  and  searcher  could  be  the  same ^  however^  in 
this  study  we  have  been  concerned  with  intermediary 
searchers y  l»e.>  information  professionals  who  perform 
searching  on  behalf  of  others. 

4.  Search  -  the  process  bf  question  analysis,  construction  of 
search  strategy,  and  cbnduct  bf  searching  in  order  to 
retrieve  Items  as  pbtential  answers  to  a  question. 

5.  Retrieved  It^ma      the  records. retrieved  from  a  database(s)  in 
whatever  form  the  database  allows  for  (e.g.,  abstracts)  and 
the  tiaer  or  searcher  specifies;  all  or  some  of  the  retrieved 
items  are  provided  to  a  user  as  potential  answers. 
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Investigation  into  the  nature  and  behavior  of  variables  in  these 
five  classes—the  information  seeking  and  information  retrieving 
varlables~i8  a  significant  area  of  study  for  at  least  four  reasons: 

1.  Cbhtributlbn  to  kno\d.ea^  ^  Investigations  into  the 
construction  and  experimental  verification  of  models 
Involving  information  seeking  and  information  retrieving  are 
^^.^^P^^^f^^.^^^^'^^-^^^^Q^  empirical  foundations  of 

information  science  in  general  and  information  retrieval  in 
partictxiari 

2-  Bestgn  of  information-syatems  ~  By  necessity  the  design  of 
ail  retrieval  systems  and  particularly  of  those  that  attempt 
to  incorporate  a  certain  a:apuht  of  ihtelligehce  ahd/6r 
natural  language  has  to  be  based, on  certain  assumptions  about 
human  infortnatidn  seeking  and  relations  between  auestlohs  and 
ansi^ers.    Mote  pfteri  than  iiot  the  source  for  these 
assumptions,  is  intuition  or  guessing »  rather  than  scientific 
evidence.    More  solid  sctehtific  evidence  on  the  cognitive 
aspects  of  information  seeking  and  searching  is  urgently 
needed  for  improvement  of  existing  systems  and  designing 
future  syistemsj  particularly  in  the  area  of  human-systems 
interaction. 

3.  Guiiatice  to  ihfbrmat ion  seekers  (in  the  conduct  of  effective 
and  efficient  searching)  -  This  includes  training  of 
information  professionals  as  searchers  and  many  aspects  of 
transparency  for  system-^ser  interfaces    The  present  state  of 
such  guidance  leaves  much  to  be  desired i    Scientific  evidence 
on  the  nature  of  the  searching  process  can  definitely  help 
improve  this  critical  areai 

4.  Relation  to  cognitive  science  and  artificial  lnte44.4gence  - 
Problems  addressed  here  are  closely  related  to  studies  of 
question  answering [ i *2 ] i    However,  as  Graesser  remarked,  the 
various  different  fields  investigating  questions  are  rather 
isolated  from  each  other  and  a  raultidisciplinary  study  of 
questioning  is  necessary.    This  was  an  attempt  to  study 
questioning  in  an  interdisciplinary  manner  as  called  for. 
this  is  especially  significant  for  expert  systems. 
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2.  AIM,  OBJECTIVES,  APPROACH 


The  aim  of  the  ctiidy^  as  mehtiohcd  in  the  preface^  was  to 
contribute  to  the  fbmal^  scientific  characterization  of  the  eiements 
ihx^blved  iii  information  seeking  and  retrieving,  particularly  in 
relation  to  the  cognitive  decisions  and  human  interactions  involved. 

_  The  objectives  were  to  conduct  a  series  of  experiiuen^s  related 
to  the: 

^*  ^9^^g.^__Q^_§^gg^l9^8  in  information  retrieval  as  provided  by 
the  user  posing  the  question 

2.  Structure         classification  of  ^ueisrtibns 

3i  Cognitive  traits  and  decision-making  of  Searchers 

4.  Comparative  nature  of  the  search  by  different  searchers  of 
the  same  question. 

 The  study  concentrated  on  applying  and  testing  the  models, 

methods,  and  measures  developed  in  the  preceeding  study  (mentioned  in 
the  Preface) c    These  together  with  some  of  the  underlying  concepts 
were  elaborated  in  a  number  of  articles  and  reports  [2-1  to  2-12]  and 
are  summarized  in  Part  II. 

The  approach  taken  was  to  use  as  real-life  situation  as 
possible,  rather  than  to  build  laboratory  systems  and  situations  and 
test  them.    This  approach  has  obvious  strengths  and  weaknesses.  The 
major  strshgth  was  elimination  of  artificialities  and  smailness  of 
scale  built  in  by  necessity  in  the  laboratory  approach^    The  weakness 
was  the  difficulty  in  imposing  controls  and  the  impbssibiiity  for 
allowing  for  some  parts  of  the  real-life  situation^  especially  where 
several  searchers  are  searching  the  same  questions.    The  users  were 
real  in  the  sense  that  they  posed  questions  related  to  their  research 
or  work  at  hand.    The  users  were  not  paid  for  their  time,  but  the 
search  was  provided  free*    The  searchers  were  real  in  the  sense  that 
searching  is  a  part  of  their  professional  function.    The  searchers 
were  paid  for  their  searches i    The  searching  ras  real  in  a  sense  that 
the  existing  databases  on  DiALOG  were  used.    The  control  was  that  all 
the  searching  was  done  under  the  same  conditions,  using  the  same 
search  tools^  equipment,  and  protocols.    No  other  restrictions  were 
placed  on  the  searchers  or  search  time. 

 However,  for  control  purposes  there  were  two  major  restrictions. 

First,  searching  for  ane  question  by  different  searchers  involved 
only  one  and  the-saim»  database*    Searchers  did  lipt  choose  the_ 
database^  project  staff  chose  the  database  according  to  the  close 
connection  between  the  subject  of  the  question  with  a  given  database. 
In  real  life  more  than  one  database  may  be  searched,  but  we  could  hot 
control  independent  selection.    Second,  searchers  did  not  have  access 
to  users  for  an  interview  about  the  questiph.    Users  provided  an 
elaborate  statement  of  the  question,  together  with  an  indication  of 
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the  type  of  search  atid  restrictions  wanted  (see  Appendix- A)  and  each 
soarcher  was  given  that  statement  •    There  was  no  way  that  we  could 
control  the  Interview  of  five  different  searchsrs  with  the  same  user* 
Besides J  the  power  of  interviewing  was  not  under  investigation.  Thus 
we  used  the  approach  of  doing  the  searching  without  an  intervlx-w^  an 
approach  that  Is  not  uncommon  in  reai-lifci  aa  well* 

While  we  lahel  bur  studies  as  experiments,  they  are  not 
experiments  in  the  narrow  and  scientific  sense  of  testing  a  randomly 
selected  sample  under  controlled  conditions  with  a  well  defined 
hypothesis.    These  are  experiments  in  a  broader  scientific  sense 
im^olving  observation  of  the  behavior  of  a  population  under 
conditions  which  were  as  c».>atrclled  as  possible.    Because  this  was 
not  a  random  ss^lection  of  uriers,  questions^  searchers^  and 
information  sources  the  power  of  statistical  conclusions  is 
different .    The  conclusions  pertain  to  the  sample Ge;n<=»rallzations 
should  be  treated  with  caution  and  as  hypotheses  for  further  testing. 
However,  the  sample  used  may  not  be  atypical  as  found  in  the  general 
population  of  information  users  and  searchers. 
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3.  RELATED  STUDIES 


3.1  Reviews 


Several  excellent  reviews  have  appeared  recently  on  topics 
related  to  this  study^  thus  our  own  review  can  be  brief •    The  most 
extensive  Is  the  review  by  Belkin  and  Vlckery  [3-1]  on  Interaction  in 
information  retrieval  systems, ^  iiave  found  a  large 

literature  dealtag  with  interactions  in  i  variety  of  systems  from 
document  retrieval  to  knowledge-based  systems,  they  also  found  that 
the  number  of  scientific  studies  Included  was  relatively  low  and 
results  not  well  cumulated.    Many  models  have  been  suggested, 
however^  the  methodological  'problems  in  testing  these  models  are 
serious i    A  major  conclusion  is  that  "•••research  has  not  yet 
provided  a  satisfactory  solution  to  the  problem  of  interfacing 
between  the  end-user  and  large  scale  databases •" 

  Because  there  are  many  factors  involved  in  htman  interaction 

with  retrieval  systems,  a  cohsldarable  amount  of  research  hao  been 
devoted  to  the  study  of  user  characteristics  in  general  and 
psychological  factors  relevant  to  human  behavior  during  user-system 
interaction  in  particular^    Dervin  and  Niles  [3-21  reviewed  the 
former  studies  and  Borgman  [3-3}  the  latter^    Dervin  and  Niles  point 
out  the  changing  paradigm  in  user  studies  and  advocate  more  objective 
studies.    Borgman,  in  concentrating  on  the  review  of  psychological 
problems,  cphcluded:      * isbme  of  the  research  results  have  advanced 
bur  theoretical  understanding  of  the  interaction  between  human  arid 
computers^  arid  in  the  process  have  added  to  the  general  body  of 
behavioral  knowledge*"    She  suggested  that  further  research  is  rieeded 
to  study  why  information  systems  are  hard  to  use  and  how  they  can  be 
improved •    She  adds  that  an  expansion  of  the  study  of  cognitive 
mbdelis  is  needed  as  well  as  studies  on  how  to  apply  results  to  system 
design  and  to  user  trainings 

In  earlier  reviews.  Bates  [3-4]  noted  a  clear  absence  of 
research  comparing  strategies  which  could  Irifluerice  searcher 
performance  and  Penichel  [3-5]  suggested  that  research  should  be 
directed  towards  understanding  the  nature  of  user-system  interaction, 
and  "iiiwhat  is  actually  happening  at  the  man-machine  interface  of 
online  systems." 

it  is  most  indicative  that  the  coricluslbri  of  ail  of  the  reviews 
of  the  literature  on  the  human-system  interface  in  general  and  online 
searching  in  particular  are  slmilari  despite  different  orientations 
of  the  reviews  and  different  backgrburids  bf  the  reviewers.  The 
literature  is  large,  there  are  numerous  mbdels,  inventories  of 
variables,  and  descrlptlbri  arid  prescription  of  processes  suggested. 
However,  there  is  still  a  remarkable  paucity  of  empirical  evidence, 
controlled  observatlpris  and  experimental  studies i    As  a  result,  as 
yet  we  do  not  have  clear  ideas  supported  by  evidence  about  such 
aspects  as: 
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i..What  cognitive  traits  have  significant  iaffect  oh  searching 
for  information  in  particular  and  on  human-system  interaction  in 
general? 

^•.How  do  people  differ  in  searching  and  in  human-system 
interface J  how  much  and  how  significantly  do  they  differ,  and 
what  accounts  for  the  difference? 

•••What  makes  a  "good"  search  and  searcher? 

-  -  The  significance  of  these  and  similar  questions  for  the  design 
of  future  and  more  intelligent  information  systems  and  human-system 
interfaces  cannot  be  overemphasized*    The  studies  addressing  these 
questions  with  supporting  evidence  are  still  in  an  exploratory  stage. 
This  study  Is  one  of  th&a. 


3.2  Models 


 A  number  of  models  have  been  suggested  involvirig  the_cbntext  of 

information  seeking,  questions,  and  the  search  process.  Theterm 
"model"  is  used  here  broadly  to  include  classic  models  as  well  as  an 
inventory  of  variables  and  descriptiphs  or  prescriptions  of 
processes.    Some  representative  models  are  reviewed  here. 

Already  mentioned  is  the  exhaustive  listof  variables  by  Fidel 
and  Soergel  [1-7] •    This  long  list  of  over  200  variables  grouped  into 
eight  broad  categories  illustrates  the  complexity  of  online 
searching. 

The  context  of  ihf brmatibh  seeking  received  considerable 
attention  in  modeling.    For  a  long  time  the  predominant  concept 
around  which  models  revolved  was  the  concept  of  information  need;  we 
shall  mention  Taylor's  work  E3-6J  as  representative  of  this  school  of 
thought.    Slowly, modeling  changed  to  that  of  problem  orientation, 
viewing  the  problem  behind  the  question  rather  than  information  need 
as  central  to  the  inforaatlon  seeking  contexts    The  work  by  Belkin 
E3-7J  Is  representative  of  the  problCToriente^  of  thought, 

which  has  Increasingly  borrowed  notions  and  approaches  from  cognitive 
science.    The  study  reported  here  belongs  in  this  category. 

The  nature  of  questions^  as  reviewed  by  Graesser  and 
Black  [1-2]^  has  been  a  subject  of  study  in  a  number  of  fields  from 
philosophy  and  logic  to  computer  scien^^  and  artificial  intelligence. 
Librarianship  has  many  works  on  classification  of  questions,  some 
going  back  over  50  years  [3-8] •    More  recently,  the  whole  area  of 
questions  and  questioning  becpie  an  intensiv^^  area  of  study  in 
artificial  intelligence  because  of  its  importance  to  natural  language 
processing^  question-answering  systems,  and  expert  systems.    The  book 
by  Graesser  and  Black  is  representative  of  work  in  this  area.    So  is 
the  pioneering  work  by  Lehnert  [3-9].    Among  other  things,  she 
provided  a  novel  classification  scheme  for  questions.    The  work  on 
questions  in  artificial  intelligence  is  innovative,  but  it  also 
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demonstrates  that  the  progress  in  this  area  is  slow  and  incremental. 
The  reported  study  is  complementary  to  this  work  in  artificial 
intelligence. 

_    A  litimber  of  works  in  information  science  have  been  devoted  to 
modeling  and  description  of  the  search  process,  these  range  from 
simple  flowcharts  to  complex  analysis  of  the  elements  and  steps 
involved.    Here  are  some  representative  works: 

•••elements  and  tactics  in  question  analysis  and  search 
strategy,  Bates  [3-10,  3-11] 

ii. types  of  search  strategies,  Markey  and  Athertbri  [3-12] 

iiidefinition  and  principles  of  user  interviews  and  search 
processes,  Soergel  [3-13] 

i . .identification  of  heuristics  and  tactics  that  are  applicable 
to  a  wide  range  of  search  problems i  Barter  and  Peters  [3-14] 

Most  of  the  descriptions  in  these  studies  have  been  inferred 
from  observations  of  professional  practice  or  describe  desires  to 
improve  practice  and  make  it  more  standardized.    Remarkably  few 
models  have  been  put  to  a  scientific  test. 


3.3  Empirical  Studies 


The  factors  affecting  online  searching  and  human-s/stem 
interface  have  been  studied  in  a  nt^ber  of  experiments  in  which  data 
were  collected  under  (more  or  less)  controlled  conditions.    Here  is  a 
list  of  representative  topics  in  such  studies: 

.«.di,^ferehce8  in  searching  and  in  search  results  as  affected  by 
various  degrees  of  searching  experience,  Fenichel  [3-15] 

...relationship  betwee^  cognitive  characteristics  or 

educational  level  of  searchers  and  type  of  searching  and/or 
search  results,  Brindie,  Beiiardo,  Woelfl  [3-16,  3-17,  3-18] 

iiitypes  of  elements,  sequences,  and  modifications  in  the  search 
process,  Penniman,  Fidel,  Oldroyd  and  Cetroen  [3-19,  3-20,  3-21, 
3-22] 

iiieffects  of  the  type  of  training  received  by  searchers^ 
Borgman  [3-23] 

...effects  of  various  types  of  search  questions  and  various  user 
goals  on  searching.  Rouse  and  Rouse  [3-24] 

Some  of  the  conclusions  resulting  from  these  studies  were: 

...modification  of  search  strategy  during  searching  is  very 
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important,  but  most  searchers  do  not  use  the  interactive 
capabiiltles  of  online  systems 

••isearcher  training  and  experience  affect  search  results ^  but 
searchers  with  little  training  often  do  quite  well 

i.. searchers  differ  considerably  in  searching  the  same  question 

•••there  are  no  conclusive  cognitive  or  psychological  traits  of 
searchers  that  are  associated  in  a  significant  way  with  search 
results. 

The  study  repored  here  is  closely  related  by  type  to  the 
empirical  studies  reviewed  above.     It  is  building  on  these  studies. 
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PART  II.  METHODOLOGY 


4.  MODEL 


4.1  Overview 


The  model  used  here  concentrates  on  these  aspects: 

1 .  Thiei  information  seekers  or  users  as  a  source  of  qaestions 
asked  of  information  systems^  and  the  context  involved  in 
terms  of:  (i 5  underlying  problem,  (ii)  intent,  (iii)  internal 
knowledge  state,  and  (iv)  public  knowledge  estimate 

2i  The  question  in  terms  of  its  structure  and  classification  in 
several  categories 

3.  The  searcher  in  terms  o£  several  cognitive  traits  as 
exemplified  by  results  from  standard  tests  and  several 
indications  of  searching  experience 

4.  The  search  in  terms  of  various  measures  related  to  structure, 
efficiency  and  effectiveness  and  the  degree  of  agreement 
among  searchers 

5.  The  items  retrieved  in  terms  of  their  evaluation  as  to 
relevance  by  the  user  and  the  degree  of  overlap  among 
searchers . 

The  various  measures  and  indicators  applied  in  the  study 
characterize  the  state  of  one  or  more  elements  In  the  model. 

4.2  information  Seekers  and  Context  of  Information  Seeking 

Quite  obviously  there  is  more  to  a  question  in  information 
retrieval  than  the  words  in  the  text  of  the  questions  The 
information  seeker  or  user  has  a  set  of  internal  cognitive  and 
knowledge  states  or  elements  rad  a  set  of 

conditions  that  affect  everything  connects  information  seeking. 

Of  the  possible  factors  explored  by  a  variety  of  investigators  (see 
literature  review)  we  have  selected  four  to  explore  their  effect: 

li  Problem^  underlying  the  question 

2i  In£en£  for  use  of  the  information 

3.  Internal  Knowledge  State^  of  the  user 

4.  - Public  KnQWle4g^  Estimate  of  the  user 

9 


30 


The  essence  of  defining  and  delimiting  the  information  seeking 
context  in  the  above  way  is  its  problem  orientation.    Probiem  solvin 
has  been  a  topic  of  research  in  cognitive  science  for  a  considerable 
time  [4-1].    Probiem  approach  coupled  with  internal  knowledge  states 
has  also  been  used  in  theoretical  and  experimental  studies  related  t 
different  information  system  designs  by  Belkin  and  associates  [4-2]. 
Furthermore^  the  problem  approach  is  explicit  in  the  informatibh 
processing  modeii  the  mainstay  of  cognitive  science  [4-3].  Thus, 
this  approach  to  modeling  of  information  seeking  has  a  wide 
?Ppi^^?^?^^_^^^l^  information  science,  cognitive  science  and 
artificial  intelligence. 

4.2.1  Problem 


Problems  facing  information  seekers  are  at  the  root  of  all 
information  retrieval  requests _•    In  other  words:    ho  problem^  no 
request.    In  the  framework  of  information  retrieval ^  a  problem  can  hi 
defined  as  .9 n  unknown  in  a  Work  or  situation.    Problem  signifies  that 
which  causes  difficulty  in  finding  or  working  but  a  solution. 

The  resolution  of  problems  requires  information  oh  the  basis  of 
which  decisions  can  be  made  and  actions  undertaken.    Such  informatiot 
can  be  obtained  in  many  ways.    One  way  is  to  obtain  or  deduce  it  fron 
thebody  of  existing  public  knowledge i    The  obvious  implication 
follows : 

information  retrieval  is  applicable  only  for  the  type  of 
problems  for  which  there  is  already  a  solution  (or  a  partial 
solution)  in  the  existing  body  of  public  knowledge  or  for  which 
a  solution  could  be  deduced  from  such  knowledge,  unless,  of 
course^  the  exercise  is  undertaken  to  prove  that  public 
knowledge  on  a  given  topic  does  not  exist. 

 To  i^arn  more  about  problems  we  can  borrow  from  research  in 

cognitive  science  [4-4].    In  problem  solving  research  a  problem  is 
said  to  exist  when  (a)  at  a  given  state,  (b)  it  is  desired  to  be  at 
another  state,  and  (c)  there  is  no  clear  way  to  get  from  (a)  to  (b). 
Thus i  a  problem  involves  givens  (a  given  state  at  the  outset),  goals 
(a  goal  or  desired  state),  and  a  mechanism  presenting  obstacles  to 
change  from  a  given  to  a  goal  state.    Either  of  the  states  could  be 
well  defined  or  poorly  defined^  leading  to  four  possible  categories: 
(i)  the  best  cases  well  defined  given  and  goal  state;  (ii)  the  worst 
case:  poorly  defined  given  and  goal  statei  and  the  cases  in  between 
(iil)  well  defined  given  and  poorly  defined  goal  state,  and  (iv)  vice 
versa. 


4.2.2  Intent 


In  submitting  a  questidh  to  a  retrieval  system  ah  information 
seeker  inevitably  has  some  purpose  in  mind  for  the  use  of  the 
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Information*    The  4atent  (in  the  framework  of  information  retrieval) 
can  be  defined  as  a  planned  or  prospective  use  of  information, 
including  cbxistraintSj  if  any^  on  that  information*    In  other  words, 
in  making  requests  information  seekers  have  some  preconceived  ideas 
about: 

•••the  use  of  the  information  in  respect  to  the  problem 

•••the  ambuht  of  time  and  effort  they  are  willing  to  spend  in 
absorption  of  or  deduction  from  the  information 

•••the  ihformatlbhal  value  characteristics  in  information 
responses  they  deem  highly  desirable:    completeness,  precision, 
reliability,  timeliness,  novelty,  confirmation,  etc^ 

f char^ateriatie  of  responses  deemed  most  desirable  as 
to  the  language,  source,  fomulation  and  the  like 

••^the  economlc-vaiue  they  attach  to  responses,  and  consequently 
how  much  are  they  able  or  willing  to  spend  on  responses  (if  not 
provided  free  of  direct  charges) 

 The  intent  in  infomation  retrieval  and  the  goal  state  in 

problem  solving  as  discussed  above  are  related  elements^  However, 
they  are  treated  separately  because  the  ihfprmatibh  seeking. intent 
can  be  a  very  specific  aspect  of  problem  solving,  exclusively  devoted 
to  the  use  of  supplied  ihfbrmatibn  within  the  broader  context  of  a 
goal  in  problem  sblving.    For  instance^  the  intent  of  an  Information 
request  may  be  to  prbvide  a  bibliography  for  a  thesis,  as  opposed  to 
the  broader  goal  of  sblvihg  the  issues  raised  in  the  thesis • 

Ihfbrmatibn  intent  cbuld  be  cbhvergeht ,  where  a  type  of  rer<ponse 
is  specified,  or  divergent ,  where  the  kind  of  responses  are 
unspecified.    The  convergent-divergent  classification  is  not  a 
dichbtbmyi  but  a  cbhtihuum.    The  Importance  of  a  recognition  of  these 
types  bf  intentis  is  because  intents  on  different  ends  of  the 
continuum  may  have  significantly  different  effects  on  performance. 


4,2.3  Internal  Knowledge  State 


People  ask  questions  because  they  don't  know  something  or  they 
want  to  confirm  something^    Either  case  deals  with  their  state  of 
knowledge*    In  the  framework  of  information  retrieval.  Internal 
khowled£;e  state  is  a  personal  variable  referring  to  the  degree  of 
knowledge       information  seeker  has  about  the  problem  at  hand  and/or 
the  request  arising  from  the  problem*    This  element  could  be  mbre 
accurately  (but  less  popularly)  described  as  the  extent  bf  an 
information  seeker's  Ignorance  about  the  problem  or  request.  Belkin 
[4"-2]  called  this  an  Anomalous  State  of  Knowledge  and  as  meritioned 
developed  an  information  retrieval  procedure  based  on  it. 

Internal  knowledge  state  involves  many  aspects  related  to 


cognitive  structures  and  processes:    how  knowledge  is  stored, 
organized,  associated^  retrieved  and  changed  in  dne^s  mind.  Research 
in  cognitive  science  is^devoted  to  these  questions  [4-3  through  4-5]* 
Unfortunately,  in  information  retrieval  up  to  now  little  use  has  been 
made  of  the  relevant  theories  or  findings  from  cognitive  science* 
The  model  here  explicitly  recognizes  this  element-    The  degree  of 
internal  knowledge  of  an  information  seeker  about  ths_prbblem  and/br 
the  request  at  hand  has  potential  for  considerable  influence  on 
search  performance  and  conduct.    At  issue  is  the  difficult  question: 
how  to  represent  a  knowledge  state  to  be  useful  for  retrieval? 


4.2.4  Publics-Knowledge  Estimate 


Public  Knowledge  is  the  recorded  knowledge  on  a  subject  in  the 
public  domain;  in  the  bphtext  of  information  retrieval  it  refers  to 
the  literature  on  a  subject  (where  * literature '  is  broadly 
interpreted  to  include  all  types  of  records  in  any  medium) i 

People  ask  questions  of  information  systems  within  the  framework 
of  public  knowledge.    This  involves  a  number  of  aspects  such  as: 
their  perception  of  what  is  (or  is  not)  in  the  public  knowledge  about 
the  problem  at  hand;  their  perception  of  (or  their  familiarity  with) 
the  organization  of  public  knowledge;  their  relation  to  public 
knowledge  about  the  problem  at  hand  (e.g^,  an  active  contributor); 
their  notion  about  what  they  can  get  from  public  knowledge,  and  so 
on. 


The  internal  knowledge  state  of  an  information  seeker  is 
measured  against  public  knowledge i    The  approach  to  information 
seeking  by  a  user  may  be  influenced  considerably  not  only  by  the 
internal  knowledge,  but  also  by  a  perception  of ,  or  knowledge  of 
existing  public  knowledge,  and /or  the  gaps,  inaccuracies,  and 
Inadequacies  in  that  knowledge. 

 Furthermore^  such  a  perception  about  public  knowledge  may 

greatly  influence  the  evaluation  of  the  effectiveness  of  the 
information  system. 


4.3  tlhar^acter  Is  tics— of  -Quest  ions 
4.3.1  Struct ure-of -Questions 


As  a  rule,  requests  in  information  retrieval  consist  of  three 
basic  elements:  a  lead-in,  a  subject,  and  a  query.    The  lead-in  is 
not  directly  searchable;  it  consists  of  phrases  such  as:     "i  am 
interested  in  informatibh  about ... "    The  subject  (or  subjects)  is 
(are)  the  central  cbncept(s)  bf  the  search.    The  query  is  the 
specific  aspect  asked  about  the  subject.    For  instance: 


12 


EKLC 


33 


What  are  the  advertising  expenditures    of    the  auto  Industry? 


ERIC 


tEAD-IN  QUERY  SUBJECT 

it  is  hard  if  not  Impossible  to  search  a  request  which  does  not 
have  both,  subject  and  query  elements  present.    But  the  structure  can 
be  more  complex.    For  Instance,  In  a  request: 

•••there  can  be  several  subjects  and/or  queries 

•••the  subject  arid/or  query  may  have  modifiers  for  more  specific 
or  alternative  concepts 

•••a  query  may  be  implied  by  the  lead-in,  in  which  case  the 
lead-in  makes  for  presupposition  of  so  called  world  knowledge 
(e^g.  ••What  are  quarks?"    Subject:  quarks;  query:  What  are  » 
definition  of). 

•••there  may  be  additional  elements  which  are  not  modifiers  but 
are  constraints  on  output  (e.g.  only  in  English"). 

The  more  complex  structure  of  requests  can  be  rejsresented  as: 

L_J  L   ]  r  ] ,  r  _j 

Lead-in  Modlfier(s)  Query(ies)  Modifier(s)  Subject(s)  Constraint(s) 
^•3iZ  Glaaalfication^  of  Queatlong^ 

It  is  not  uncommon  to  hear  searchers  describe  given  questions 
as^    "complex,"  "specific^"  "very  general,"  "difficult," 
"unsearchable,"  "unclear,"  and  the  like^    In  such  cases  searchers  are 
(possibly  even  unvUllngly)  applying. certain  general  attributes  to 
classify  questions.    While  ho  generally  accepted  claesificatibn  of 
requests  in  information  retrieval  exists^  it  Is  of  practical  interest 
to  specify  certain  attributes  which  could  be  used  to  classify  or 
describe  requests.    Fivesuch  attiributes  are  suggested  here  by  which 
requests  could  be  classified. 

!•    DOMAIN:    The  general  subject  area  or  topic  of  the  request. 
Categorization:    by  subject;  done  on  the  basis  of  some 
existing  subject  classif icatioui 

2.    CLARITY;    The  degree  (or  lack)  of  ambiguity  and  possibility 
for  more  than  one  interpretationi 

Categorizationt    clear /unclear;  done  on  the  basis  of 

ambiguity  In  respect  to: 

!•  semantics  "leaning  of  terms  in  request 
2.  sjnatax  -  logical  relation  between  terms 
3j,  context  -  the  problem,  intent,  and  internal 
knowledge  state  underlying  the  request. 
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3.    SPECIFiem;    The  hierarchical  level  in  the  meaning  of  terms 

whole  request; 
Cat egortzatibn;    f r om  high  to  low  specificity;  dpiie  by 
categorizing  tertas  on  a  continuum  from  very  specific  (object 
language)  to  very  general  (meta  language): 


META  META 
OBJECT  OBJECT 


QUERY  SUBJECT 


4.  XIQ^gl^EXITY;    The  number  of  search  concepts ^  their  modifiers 
and/of  constraihti  in  a  request. 

Ca^egorizatibn;    from  high  to  low  complexity;  done  in  terms 
of  cbuhtihg  of  search  concepts  and  possible  permutations. 

5.  gRESUPPOSITIDN;    The  implied  (not  expiicit)  conceptual 
aspects  derived  from  sharing  of  cotton  sense  or 
wbrld-^howledge  Indicated  by  linguistic  devices;  For 
instance  "What  Is .. ."implies  request  for  definition;  "Where 
Is..."  is  a  request  for  location;  ''When  was.,,"  is  request 
ihvblvihg  a  time  element ^  etci 

Categorization;  most  common  conceptual  presuppbsition 

in  information  retrieval  requests  involve:  (1)  existence  or 
\rerification,  (ii)  identity  or  definition,  (ill)  quality, 
(iv)  relation,  (v)  number;  (vi)  location,  and  (vii)  time; 
done  on  basis  of  e>:amination  and  applidation  of  world 
knowledge i 

We  have  selected  to  study  two  aspects: 

!•  The  consistency  of  Judges  in  determining  the  structure  of 
questions  and  in  categorizing  qtiestlbns  fbr  clarity^ 
specificity,  complexity  and  presupposition 

2.  The  effect  of  these  characteristics  on  search  performance. 
4.4  Searchers 


What  factors  have  asigni  leant  impact  on  searchers*  decisions 
and  thus  on  retrieval  effectiveness  and  efficiency  as  well?  Tlte 
ansi^er  is  not  yet  clear.    Several  studies  reviewed  above  have 
addressed  the  question i  but  there  have  been  too  few  studies  to 
provide  fully  tested  answers.    Fur^thermore,  several  dozen  papers 
about  the  tbplchave  been  written,  extrapolating  from  experience  and 
cbnmibn  sense.    Derived  from  these  studies  and  papers,  here  is  a 
sunnhary  bf  elements  that  have  a  significant  impact  on  searchers' 
decislbns: 

*Oygahizational  environment:  nature  of  the  institution;  nature 
of  information  seekers*  group;  managmeent  policies;  technical 
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aspects  such  as  setting,  tools,  and  reisources  available* 

t nenc tal  -considerations :    pricing  poiiciet^  of  database 
producers,  and  vendors;  type  of  charging  or  cost  recovery 
policies  In  the  organization;  cost, consciousness •  (Being 
overly  obsessed  with  costs  generally  leads. to  a  decrease  In 
retrieval  performance  and  thus,  paradoxically  to  lower  cost 
ef  f  ectiv*»^-CSd  • ) 

*ltrfogmatirCm  retrieval  knowledge  and  competency:     among  the  most 
important  are: 

•••degree  of  knowledge  about  Information  retrieval  systems 
and  processes  in  general 

•••familiarity  with  specific  databases  and  their  tools; 

•••competency  in  the  mechanics  of  searching,  €•§•, 
commands,  use  of  technology  involved; 

•••type^  breadth  and  depth  of  education  and  training 

*Sub jec t JSiowledge :  degree  of  knowledge  about  the  subject  being 

searched,  particularly  the  subject  vocabulary  and  information 
soruces.    Knowledge  of  language:     semantic  associations, 
syntactic  tranformations,  writing  norms  and  styles • 

*Gognitive  Jactora:    a  variety  of  factors  which  have  a  positive 
effect  on  retrieval  performance  and  level  of  effort  are: 

iiiability  to  make  linguistic  associations; 

•  ••ability  to  reason  .symbolicaily; 

•  •inability  to  think  iii  a  logical  and  analytical  manner • 

While  recognizing  all  of  these  factors  in  this  study  we  have 
concentrated  only  on  a  certain  set  of  cognitive  characteristics  of 
the  searcher  and  to  some  extent  oh  the  subject  knowledge  of  the 
searcher,  as  they  effect  effectiveness  and  efficiency  of  retrieval. 

We  have  selected  to  explore  the  effect  on  search  performance  of 
the  following  cognitive  traits  of  searchers: 

1.  the  ability  to  make  inductive  inferences  through  woird 
association 

2.  the  ability  to  make  deductive  inferences  through  symbolic 
reasoning 

3^  the  style  or  mode  in  acquiring  and  using  information  in 
problem  solving  and  learning 

Clearly,  these  are  not  the  only  ccignitive  triits  that  may  be  of 
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Importance  in  searching.    They  have  been  selected  as  plausible 
candidates.    The  whole  area  of  investigation  has  t5arely  begun,  thus 
any  selection  of  traits  Is  explorativra  in  nature. 


4,5  Search 


There  are  a  number  of  ways  that  the  elements  or  subprocesses  in 
a  search  have  been  described  and  classified.    A  common  way  is  to 
distinguish  among: 

!•  Questtbn  analysia;  procedures  that  deal  with  decisions  oh 
terminological,  semantic  and  pragmatic  (cbhtextual)  aspects 
of_?_qaestion  in  preparation  for  searching,  including 
determinetion  of  appropriate  information  sources. 

2-  Search^atrategy;  procedures  that  deal  with  decisidhe  on 
syntactical  and  logical  aspects  of  searching^  incorporation 
of  constraints,  if  any,  and  determination  of  appropriate 
tactics  related  to  a  desired  level  of  effectiveness  ahd/br 
ef  f  icieticy. 

3.  5earchlng^:  the  physical  conduct  of  the  search,  including  the 
system,  using  appropriate  protocols  and  obtaining  the  output. 

4.5.1  Question  Analysis 


Question  analysis  involves  a  set  of  procedures  by  which: 

...possible  ambiguities  In  a  request  are  clarified 

• • .search  concepts  are  Identified  and  expanded  upon 

...If  possible  and  necessary  an  Interview  or  negotiation  is 
conducted  with  an  information  seeker 

. ..and  the  search  concepts  are  translated  into  terminpiogy 
acceptable  to  (or  dictated  by)  a  given  database(s).    This  also 
Implies  a  previous  selection^of  database,  if  necessary. 

In  this  study  we  have  concentrated  on  two  aspects: 

\,  Selection  of  terms  for  searching  and  the  degree  of  agreement 
In  term  selection  by  different  searchers  searching  the  same 
9?^^??^^ J'^s®^        the  written  question  statement  and  using 
any  tools  as  desired. 

2.  Differences  among  the  search  statements  based  on  different 
sources  for  search  term  selection^  namely: 

(1)  from  a  tape  recorded  problem  statement  by  users,  but 


yyithoot  recourse  to  the  written  question  statement 

(ii)  from  a  taped  problem  statement  and  the  written  question 
statement 

(iti)  from  the  written  question  statement  using  only  the 
words  in  the  question  as  search  tems  without  any 
further  elaboration 

(±v)  from  the  ^rritten  question  statement  plus  terms  from  an 
appropriate  thesaurus  for  elaboration 

Aspect  !•  was  done  by  "outside  searchers"  and  aspect  2*  as  the 
four  "project  searches"  (i)-(iv). 


4,5»2  Searchh Strategy 


As  mehtiphed^  question  analysis  deals  with  semantics,  pragmatics 
and  source  selection  while  search  strategy  deals  with  syntax,  logic, 
constraints^  and  tactics.    The  end  result  of  question  analysis  is  a 
set  (or  sets)  of  search  teiis  appropriate  for  a  question  and 
databaseCs).    The  end  result  of  search  strategy  is  a  search  statement 
(or  statements)  incorporating  appropriate  logic  and  possible 
cbhstraihts^  and  oriented  to  some  level  (or  levels)  of  performance* 
The  process  could  be  dynamic  in  that  during  the  search  any  aspect 
(terms,  logic,  constraints,  tactics,  etc.)  may  change  based  on  some 
type  of  feedbacks 

 Search  strategy  involves  for  a  given  search  statement  the 

selection  of: 

^*    ^ogic ;    connections  between  terms  expressed  by  Boolean 
operators  (AND,  OR,  NOT). 

2.    Scope:    tba  number  of  classes  of  search  terms.  Determines 
which  of  the  originai  search  concepts  are  to  be  used  In  a 
given  search  statement. 

3»    Exhaus^vi^y ;     the  number  of  elaborated  search  terns  within 
each  class  ft    Detemines  which  of  the  related  terms  are  to  be 
used  for  a  search  term. 

^»    Constraints y  if  any:     the  elements  of  a  request  not  related 
to  terminology  and  specific  to  non-subject  features  of  a 
given  database  (language^  years,  source,  etci) 

In  additibh,  search  strategy  involves  selection  of  search  tactics. 

In  the  infbrmatibh  retrieval  framework  search-tactics^  refer  to 
the  specif icatlon  of  a  sesarch  statement  in  correspondence  to:  (i)  a 
desired  level  of  performance,  (ii)  the  inherent  nature  of  a  request 
and  its  context,  and/or  (ill)  prescriptions  and  restrictions  imposed 
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by_a  specific  data^rse^    Tactics  are  influenced  by  an  estimate  of  the 
effect  oh  performance  of  given  elements  incorporateid  in  a  search 
statement.    Tactics  are  also  infiuenced  by  an  estimate  of  the  'best* 
way  to  approach  the  search  for  a  request* 

Three  types  of  search  tactics  are  prevalent  in  contemporary 
jpractice: 

!•    ??c?i^s_9^i?nted  toward  achievement  of  certain  levels  of 
recall  or  jgreciaioni 

2.    Tactics  oriented  toward  ^cles  iii  searching  which  are  most 
appropriate  for  a  given  type  of  request ^  that  is^  toward 
separating  and  combining  discrete  elements  of  a  search 
statement  and  using  different  types  of  commands  during  the 
search • 


3.    Tactics  dictated  by  the  nature  of  a  specific  database  in 
relation  to  a  given  request • 

The  three  types  of  tactic!  can  be  used  in  conjunction  with  each 
pther^  they  are  hot  at  all  mutually  exclusive.    In  all  fairness,  a 
fourth  type  of  cohtemporary  tactic  should  be  mentioned:    the  random 
tactic.    These  are  tactics  used  without  any  conscious  effort  or 
ratiohale  for  ah  expected  search  performance. 

Ih  this  study  we  have  selected  to  study  the  performance  of  given 
searches  fbnnulated  by  different  sear c^       as  a  whole,  as  indicated 
by  effectiveness  and  efficiency  measures.    In  particular  we  have  also 
been  interested  in  effect  of  cycles. 


4.5.3  Searching 


 As  mentioned,  searching  involves  the  physical  conduct  of  the 

search,  including  the  establishment  of  contact  with  a  given 
i?_^P^^^i9^  sy3^®^>  ^s^^f  the  equipment ,  appropriate  protocols^ 
downloading  if  necessary,  etc.    While  we  have  recbghized  that  the 
physical  conduct  of  a  search  involves  a  number  of  variables,  we  have 
chosen  not  to  study  any  of  them •    We  kept  searching  (ehvirbhment, 
equipment,  protocols,  etc.)  constant  for  all  searchers^  except  for 
variations  in  DIALOG  system  response  time  which  was  beyond  our 
control. 


4.6  l^^a  Retrieved 


Output  from  a  search  may  be  called  by  a  number  of  names: 
references^  answers^  documehts,  abstracts^  displays,  etc.    We  have 
choseh  a  neutral  label,  "items  retrieved,"  to  designate  the  output 
for  a  search,  that  is,  the  individual  records  retrieved  from  a 
database  in  response  to  a  question i    While  databases  provide 
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different  format  bptibhs  for  each  item  retrieved^  we  have  chosen  to 
use  the  full  database  record  of  each  and  every  ite^i  retrieved.  Thus, 
"items  retrieved*'  were  full  records >    In  most  of  the  databases 
searched  for  the  project^  full  records  included  bibliographic 
ihfbrinatibu:^  index  terms  and/or  ciassifl^  and  an  abstract  • 

This  policy  resulted  In  the  most  accurate  possible  user  evaluations 
and  built  a  database  of  5411  evaluated  items  retrieved  in  machine 
readable  form. 

 Clearlyjj  the  choice  from  among  formats  available  in  various 

databases  could  be  a  variable  affecting  decisions  and  evaluations  by 
searchers,  but  we  have  chosen  not  to  study  these.    We  have 
concentrated  on  the  relevance  of  items  retrieved  as  judged  by  the 
users  and  on  the  dependence  of  the  probability  of  relevance  on  other 
variables  as  described  above  and  as  further  elaborated  iv  the  next 
chapter* 
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5  .  (SASUKBS  ASb  INDICAfORS 


5.1  Overview 


Not  only  are  the  variables  under  observation  In  this  study  very 
difficult  to  measure »  but  the  measures  themselves  are  a  matter  of 
considerable  concern.    For  the  most  part,  measures  of  performance  and 
indicators  of  the  state  of  any  entity  are  not  standardized  nor  are 
the  criteria  for  basing  the  measures  fully  agreed  upon.    With  this  in 
mind,  we  have  used  meiasures  that  have  beed  widely  applied,  such  as 
recall  and  precision.    For  variables  that  have  hd  universally  _  . 
associated  measures  or  indicators  we  have  developfsd  bur  own.  In 
these  cases,  we  have  used  a  LUcert-type  scale  with  five  points  to 
indicate  the  degree  of  agreement^  likelihood^  or  least  to  most 
some  property. 

System  performance  or  processes  are_geherally  evaluated  for 
effectiveness  (how  well  did  a  system  perform  in  respect  to  what  it 
was  designed  to  do)  and  efficiency  (at  what  costs  arid  iri  what  amount 
of  time  did  it  operate).    We  measured  both  ef f ectiveiiess  and 
efficiency^  however,  wei  concentrated  on  effectiveness. 


5.2  indicators-^f  Inf ortnatioM  Seekj^jg-Coni^axt 

Five  Likert-type  scales  have  been  used  to  obtain  an  indication 
of  the  itiforraatidn  seeking  context  first  from  users  and  then  also  as 
perceived  by  searchers; 

1.  PROBLEM  DEFINITION  SCALE 

PPi^i?^.^^^  9?  ^_?*^^i^_fE95_^  ^9  ?  would  you 
describe  you  problem  as  weakly  defined  or  clearly  defined, 
with  i  being  weakly  defined  and  5  being  clearly  defined." 


2.  INTENT  SCALE 

"Oh    scale  from. 1  to  5^  would  you  say  that  your  use  of  this 
Irif brmatibh  will  be  bpeh  tb  many  avehiies^  br,  fbr  a 
specifically  defined  purpbse,  with  1  repre^eT.tihg  open  to 
many  avenues  and  5  representing  a  specifically  defined 


weakly  defined 


2 


3 


4  5 
clearly  defined 
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1  2  3  4  5 

open  to  many  purpose  is  narrowly 

avenues  defined 

3.  INTERNAL  KNOWLEDGE  SCM.E 

"On  a  scale  from  1  to  5^  how  would  you  ranfc  the  amount  of 
knowledge  ybu  possesis  in  relation  to  the  problem  which 
motivated  this  request." 


_     1        2        3        4  ;i 

little  personal  considerable  personal 

knowledge  knowledge 

4.  PROBLEM-POBLie  KNOWLEDGE  SOME 

"Oh  a  scale  from  1  to  5 ^  how  would  you  rank  the  probability 
that  information  about  the  problem  which  motivated  this 
research  question  may  be  found  in  the  literature." 


 i  2  3  4  5  _ 

highly  improbable  highly  probable 

that  it  exists  that  it  exists 

which  these  indicators  were  recorded  are  Forms  7, 
8,  9  and  id  as  reproduced  iti  Appendix  H. 

In  addition  the  user  was  asked  on  the  question  request  form 
(Form  5): 

1.  Do  you  want  a  precise  or  broad  search? 

3  -  precise 

4  -  broad 

2.  Type  of  application  of  this  research: 

5  -  Undergraduate  study  8  -  Industrial 

6  -  Graduate  study  9  -  General 

7  -  Faculty  research  10  -  Other  (please  specify) 

3.  Do  you  want  to  place  restrictions  on  the  language  of 
publication  of *  the  articles  retrieved? 

11  -  English  only 
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12  -  Any  language 


4,  Do  you  want  to  restrict-  the  years  of  publication  of  the 
articles  retrieved? 

13  -  Last  5  years 

14  -  No  limits 

15  -  Other,  specify  19_  to  19—- 

5.  If  you  are  familiar  with  the  DiAtOG  databases,  please 
indicate  those  that  would  be  appropriate  for  your 
question  16    17   18  


5*3  Indicators  o£-Ques^og^  Structure  and^Xlassificat ion 

 In  the  analysis  for  strueture  of  the  questions  the  following 

function  components  were  used: 


(tead-in)  (Modifiers)  (Query  or  Queries) 


(Modifiers)  (Subject (s)) 


(Constraints) 

An  example  of  this  structure  applied  to  a  specific  request  follows: 

What  are  the  advertising  expenditures  of  the  automobile  industry? 

(lead-in)        (modifier  -  query)  (modifier  -  subject) 

In  the  classification  of  categories  the  following  indicators  and 
Likert-type  scales  have  been  used: 

A.  DOMAIN: 

Indicate  the  number  of  DIALINDEX  categories 

B.  etARiTY: 

li  Semantics  (meaning  of  terms) 

:   :  _; —  *   ;  . 

0  ~5 
unclear  clear 


22 


•2.  Sjmtax  (relation  between  terms) 

?   :  :  ;  ;   : 

0  —5 

unclear  clear 
C.  SPECIFICITY^ 

query  1,  etc. : 


0 

met  a 

subject  1^  etc.: 


0 

meta 
D.  COMPLEXITY: 

Nu!i:ber  of  search  concepts 
Number  of  constraints   


E.  PRESUPPOSITIONS: 

Total  number   


Number/percent  transferable  to 
the  search  process  -  -  -  

These  indicators  are  reproduced  in  Form  11*  Appendix       as  they 
were  used  by  the  searchers. 


5.4  Measures  of  Searchers 


5.4.1  Cognitive  Tests 

To  study  th€5  three  cognitive  traits  as  indicated  above 
(inductive  Inference,  deductive  Inference^  and  learning  style)  three 
tests  were  administered  to  each  searcher  (further  described  in 
Chapter  9): 

1.  Remote  Associates  Test;  measures  the  ability  to  make 

Inductive  J^Serences^  spec^  infer  the 

missing  members  of  word  setsi  Given  sets  of  three  stimulus 

wbrds^  subjects  are  required  to  fill  in  the  fourth  and 
missing  member  of  each  word  set  by  inferring  some  property, 
element,  or  characteristic  that  the  stimulus  words  have  in 
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5 

object 


5 

object 
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cdmmdn.    The  test  ihstruiheht  used  was  developed  by  Mednick 
and  Mednick  [5-i]  and,  has  beeii  applied  and  tested  for  15 
years.    In  essence  this  Is  a  word  association  test. 

2.  Symbolic  Reasoning  Test;  measures  the  ability  to  make 
deductive  inferences  based  on  symbols.    Each  item  in  the  30 
item  test  specifies  a  relationship  of  A  to  B  to  €  and 
requires  a  'true,'   'false,*  or  *d6n't  know'  conclusion  about 
the  relationship  between  A  and  6,  e.g., 

A  >  B  <  e      therefore      A  >  6      :      i     |_  bon't-Knor^ 

The  Symboli^'^easoning  Test  is  test  10  of  a  larger  battery  of 
^9_^®^^_^^9^  tests  called  the  Employee  Aptitude  Survey 
developed  by  Ruck  and  Ruck  [5-2].     It  has  been  applied  and 
tested  for  20  years.    In  essence,  this  is  a  test  involving 
inequaitttes^ 

3.  Leaacning  Style  Invaatdgy!  designed  to  measiire  ah  individual's 
preference  for  each  of  the  four  basic  modes  of  learning: 
Concrete  Experience;  Reflective  Observation;  Abstract 
Conceptualization;  and/or  Active  Experimehtatibn.    Each  mode 
represents  a  characteristic  method  of  acquiring  and  using 
infomatioh  in  learning  and_  solving  problems.    The  respondent 
describes  his/her  style  of  leaxming  by  ranking  sets  of  four 
words  representing  each  of  the  basic  abilities,  such  as  in 
this  example: 

i  iearn  by:  Feeling  -  Watching  -  Thinking  -  Doing 

The  test  instruments  to  be  used  are  those  by  Koib  [5-3] i  The 
inyeritbry  has  been  applied  and  used  for  10  years i    In  essence 
this  test  places  a  participant  in  a  grid  of  learning  styles 
and/or  their  combinations. 

The  test  instrument  cannot  be  reproduced  here  because  they  are 
copyrighted  and  sold  on  request. 


5.4.2  Searcher  Experience 


A  searcher  questionnaire  (Form  3  Appendix^  H )  contained  questions 
reflecting  a  searcher's  experience  in  online  searching: 

"1.  How  of ten  do  you  search  DIALOG?  (Please  circle  the  best 
estimate. ) 

5-baiiy,  4-Twice  a  week,  3-dnce/week,  2-Twice/mbnth,  l-Less 

2.    Refer  to  "List  A  -  Selected  Databases"  (attached)  and  please 
indicate  those  databases  that  you  search  mbst  bfteh^  in 
order  of  decreasing  use.    Belbw  each  database  code  indicate 
how  often  you  search  it,  using  the  same  codes  as  in  Question 
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2  above. 


Databases  used  most  often 
CODE:       ;     ;     ;  etc. 

3.  Frequency  of  use  of  the  above  databases. 

 l_  etc. 

4.  Now  please  refer  to  ^'tistB  -  Thesauri"  (attached)  and 
indicate  those  most  important  to  you  when  you  search. 


CODE: 


5.4.3  Measure  of  Overlap  in  Sb4 

 This  measure  indicates  for  each  pair  of  searchers  the_ degree  of 

agreement  or  overlap  in  selection  of  search  terms  in  searching  of  the 
same  question.    However ^  because  there  may  be  differences  in  how 
Searcher  1  agrees  with  Searcher  2|  from  how  Searcher  2  agrees  with 
Searcher  1  the  measure  is  asymetrlcal  (e.g.»  Searcher  1  uses  two 
terms^  arid  Searcher  2  uses  six  terms;  the  two  terms  of  81  are  also 
used  by  S2,  but  S2  has  four  more  terms i  thus  SI  is  in  100%  agreement 
with  S2,  but  S2  is  only  in  33%  agreement  with  Sl)i    The  overlap  or 
agreement  measures  for  search  terms  are: 

Sj^2    "  ^2!  °*    no.  of  search  terms  in  coSon 


jSjj  total  nbi  of  terms  used  by  Searcher  i 

^2,         "  j^l'^^21  *  search  terms  in  common 

jS2j  total  noi  of  terms  used  by  Searcher  2 

These  measures  can  be  used  to  construct  a  matrix  for  a  whole  group  of 
searchers  to  study  the  distribution  of  overlap  in  search  terms  for 
the  group  as  a  whole. 

5.4i4  Measures-  for  Overlap  in  Output 


 This  measure  indicates  for  each  pair  of  searchers  the  degree_bf 

overlap  in  retrieved  items  for  the  search  of  the  same  question.  The 
overlap  measure  for  output  parallels  the  measure  for  the  overlap  or 
degree  of  agreement  in  search  terms.    It  is  calculated  in  the  spie 
way  except  that  either  the  total  ntimber  of  retrieved  items  or  else 
the  number  of  relevant  items  retrieved  is  substituted  for  the  number 
of  search  terms.    Since  the  formula  is  the  same  as  shown  in  the 
preceeding  section,  it  is  hot  repeated  here.    Both  overlap  measures 
are  asymetrical  and  both  are  used  by  arranging  data  into  a  matrix. 
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5i5  Measurea  of  the  Seargh^^is^  a  Whdie 


5.5.1  Criteria  for  Effectiveness  Measures 


two  of  the  most  often  used  criteria  for  evaluating  the 
effectiveness  of  a  search  have  been  used: 

1.  Relevance:  the  degree  of  fit  between  the  request  and  the 
retrieved  item.    The  criteria  of  "aboutness"  is  used. 

2.  Utility  or  pertinence:  the  degree  of  actual  usefulness  of 
answers  to  an  information  seeker.    The  criteria  used  is  the 
value  to  the  Information  seekers 

_        In  this  study  both  relevance  and  utility  have  been  established 
by  the  users.    Relevance^  as  defined,  involves  judging  the 
relatedhess  of  the  question  with  that  of  each  retrieved  item 
provided.    Utility  on  the  other  hand  involves  judging  the  degree  of 
satisfaction  with  the  total  search  (i.ei,  with  all  items  provided 
collectively)  or  the  degree  of  aggregate  impact  of  the  search  results 
on  resolution  of  the  problem,  and  the  fit  with  intent  and  internal 
knowledge • 


5.5.2  Definition  of  Relevance 


Concerning  the  judgement  of  relevance  the  following  instructions 
and  definitions  have  been  provided  to  the  users  (Form  14, 
Appendix^  E) : 

"Each  abstract  should  be  evaluated  according  to  its  degree  of 
relevance  to  the  question  you  submitted  for  searching.  The 
degree  of  relevance  should  be  determined  using  the  following 
three  pditit  scale: 

RELEVANT  -  Any  dbcumeht  which  oh  the  basis  of  the 
itif drmatiori  it  conveys  is  considered  to  be  related  to  your 
question^  eveii  if  the  information  is  outdated  or  already 
familiar  to  you. 

PARTIALLY  RELEVANT  -  Any  doctment  v^ich  on  the  basis  of  the 
information  it  conveys  is  considered  only  somewhat  or  in 
some  part  related  to  your  question  or  to  any  part  of  your 
question. 


NONRELEVANT  -  Any  document  which  on  the  basis  of  the 
information  it  conveys  is  not  at  all  related  to  your 
question." 
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For  the  data  analysis  relevant  and  partially  relevant  items  were 
combined  iiitd  one  class  and  hbnrelevant  items  were  left  in  the  other. 
However,  we  provide  raw  data  for  each  question,  thus  recalcaiation 
with  any  other  combination  can  be  done. 


5.5.3  Recall  ^nd  Precision 


These  are  measures  based  on  the  relevance  judgement  of  users 
where: 

Precision  »  probability  that  a  retrieved  item  is  relevant 

Recall  »  probability  that  a  relevant  item  in  the  file  is 
retrieved 

These  probabilities  are  estimated  as  fdildW3  for  a  given  search: 

Precision  «  Ko^-^f  relevant^  items  retri^sved  by  the  search 
Total  no,  of  items  retrieved  by  the  search 

Recall  "    N4>>  of  r^levaat^^ems  retrieved  by  the  search 
Total  no.  of  relevant  items  in  the  union  of 
items  retrieved  by  all  searchers  for  that  question 

Precision  is  easy  to  establish, directly  from  the  output  of 
evaluated  items  for  a  search.    Recall  is  hot  easy  to  establish, 
because  it  is  never  apparent  how  many  items  Ih  a  file  are  relevant  to 
the  question.    Each  question  was  searched  by  a  number  of  searchers 
and  types  of  searches.    A  union  of  retrieved  items  from  all  searches 
for  the  question  was  established  (i.e.,  by  merging  all  the  outputs 
and  elimihatiiig  duplicates)  and  sent  to  the  user  for  evaioation.  In 
this  way  the_ evaluated  items  from  the  union  served  as  the  benchmark 
of  individual  searcher  recall.    This  presents  a  comparative  rather 
than  absolute  measure  of  recall  performance  for  any  given  search. 

5.5.4  Utility  Based  Measures 


These  are  measures  based  on  users  expression  of  degree  of 
satisfaction  and  value  of  the  retrieved  items  as  a  whole.    Recall  and 
precision  are  universally  used  measures.    Unfortunately,  there  are  no 
such  universalij^ used  utility  measures,  thus  we  had  to  etablish  our 
own*    The  following  questions  were  posed  to  the  users  (Form  12, 
Appendix  E)^  Which  reflect  utility  based  measures  5 

Sow  much  time  did  you  spend  reviewing  these 
abstracts?  


2.  in  an  overall  sense,  if  you  were  asked  to  assign  a  dollar 
value  to  the  usefulness  of  this  entire  set  of  abstracts  to 
you,  what  would  that  dollar  value  be? 


27 


I  cannot  assign  a  dollar  value 


3.  Gould  you  rate  your  participation  in  this  project  and  the 
information  which  resulted  as: 

5  Worth  much  more  than  the  time  it  has  taken 

4  Worth  somewhat  more  than  the  time  it  has  taken 

3  Worth  about  as  much  as  the  time  It  has  taken 

2  Worth  less  than  the  time  it  has  taken 

1  Practically  worthless 

4.  PROBLEM  RESOLUTION  SCALE  -  On  a  scale  of  1  to  5,  what 
cbhtributibn  has  this  information  made  to  the  resolution  of 
the  problem  which  motivated  your  question? 

•  :  :  ;   ;  : 

1  ^5 

nothing    substantial 

contributed  contribution 

5.  SATISFACTION  SCALE  -  On  a  scale  of  1  to  5,  how  satisfied 
were  you  with  the  results  of  the  search? 

'   :   t  :  — ;  : 

1  ^5 

dissatisfied  satisfi.ed" 


5.5.5  Ef flclengy- Measures 


 As  mentioned,  efficiency  depends  oh  the  cost  and  time^  or  the 

level  of  effort  used  to  perform  a  task  or  function.    Costs  cannot  be 
directly  measured  in  ah  experimental  setting  in  a  meaningful  way. 
However,  time  and  effort  levels  can  be  measured.    When  level  of 
effort  measures, are  available*  cost  can  be  estimated  by  applying 
appropriate  unit  cost  figures  for  time  of  personnel^  computer  connect 
time,  humber  of  printed  ahswers^  and  so  forth. 

The  fbllbwlhg  efficiency  or  level  of  effort  measures  have  been 
used  for  bnline  searches: 

1.  Number  of  command^  used  by  a  sciarcher 

2.  Number  of  command  cycles  used  by  a  searchers    A  cycle  is  a 
set  of  commands  in  sequence  from  those  used  to  select, 
combine^  and/or  expand  terms  to  a  command  used  to  type  (or 
view)  the  resultsi    A  cycle  ends  with  display  (type,  print) 
of  intermediate  or  finai  Items  retrieved  for  a  set  of 
preceeding  commands i 

3.  Number  of  search  terma  used  by  a  searcher  searching  a 
question 


28 


49 


4.  Online  conaegt:^  tttae  usied  by  a  searcher  in  searching  a 
question 

5.  Preparation^ tline  used  by  a  searcher  in  preparing  a  search  for 
a  question 

6.  Total  time  used  by  a  searcher  (connect  time  plus  preparation 
time) 


5.6  Measures  for  Items  Retrieved 


The  following  simple  measures  of  quantity  were  used: 

!•  Total  number  of  items  retrieved  for  a  question  by  all 

searches  consisting  of  the  union  or  merged  set  (duplicates 
eliminated) 

2.  totai_number— of  items  evaiuatad  by  a  user  for  a  question.  As 
explained  in  the  section  on  procedures,  when  the  size  of  the 
union  of  items  retrieved  for  a  question  exceeded  150,  only 
the  first  150  were  sent  to  the  user  for  evaluation,  thus  the 
total  number  of  evaluated  items  is  lower  than  the  total 
number  of  retrieved  items 

3.  Number  ^  items  retrieved^ and  judged  relevant  for  a  question, 
as  evaluated  by  the  user 

4.  N^ber  of  items-retrieved  and  judged  partially  relevant  for  a 
question 

5.  Ifamber-gf  items  retrieved  and  judged  nonrelevant  for  a 
question  . 

6.  Overall  precision  -the  percentage  of  relevant  plus  partially 
relevant  items  based  on  the  total  number  of  evaluated  items 
for  a  question 

These  quantities  have  been  used  to  calculate  the  chance  of 
retrieval  of  a  relevant  or  partially  relevant  item  as  a  function  of 
some  other  variable.    Thus,  these  simple  measures  of  quantity  when 
correlated  with  users,  searchers^  and  search  variables  considerably 
expanded  the  range  of  observation  beyond  the  rather  restrictive 
recall  and  precision  measittres  defined  for  searches  alone. 

We  are  able  to  examine  the  pooled  data  (involving  the  .'ium  of  the 
items  retrieved  for  ail  searches  without  elimination  of 
duplicates— this  amounted  to  8956  retrieved  items)  and  study  the 
influence  of  any  variable  on  the  chance  that  a  retrieved  item  will  be 
relevant  or  partially  relevant. 
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6.  PROeESURES 


6*1  Haara »-  QuestAoasT-EvalAiatlons 


On  the  basis  of  advertising  and  mass  mailings  about  the  project 
(see  Appeadi^),  apjproximately  80  potential  users  responded  and  40 
were  selected  to  participate  in  the  project.    Each  of  the  40  users: 

.. .received  a  form  (Form  4,  Appendix  H)  with  a  short  description 
of  the  project  and  a  questibhhaire  about  their  availability 

•••was  asked  to  submit  one  question  and  to  fill  out  a  ^uas^l^n 
Request  Form  (Form  5)  to  record  in  writing  their  question 
statement  and  to  Provide  information  about  desired  search 
constraints  (e.g.,  year  of  publication) 

•• .participated  in  a  tape  recorded  interview  conducted  by  one  of 
the  project  searchers  in  which  the  user  described  the  problem 
underlying  the  question  and  the  use  intended.    The  tapes  were 
not  transcribed,  but  were  used  as  is  for  project  searches 
(described  below). 

vindicated  the  context  measures  during  the  interview  (Forms  7 
and  8) 

ii. after  searching,  obtained  a  printout i  in  duplicate^  of  up  to 
150  items  retrieved  for  evaluation.    They  indicated  their 
relevance  judgement  for  each  iterai  returned  the  original  and 
kept  the  copy  for  their  own  use 

.filled  out  a  questionnaire  oh  Evaluatibh  of  Answers  (Form  12) 
which  among  others  contained  the  utility  measures 

To  insure  consistency  of  the  process,  the  procedures  for 
question  handling  arid  for  iriteryiews  with  users  were  specified  in 
writing  (see  Procedure  No.  1  and  2,  Appendix  I). 

Ail  the  users  completed  all  their  tasks  from  start  to  finish. 
Thus  user  response  was  100%.    The  users  were  not  paid  for  their 
effort,  but  they  received  the  search  free  of  charges 

The  40  questions  submitted  by  the  users,  together  with  their 
iridicatiori  of  desired  search  constraints  are  reproduced  in 
Appendix  A. 


6.2  Searchers 


A  mass  mailing  about  the  study  was  sent  to  the  members  of  the 
Northern  Ohio  Chapters  of  the  American  Society  for  Information 
Science,  Special  Libraries  Association,  and  the  Online  Group. 
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Presentations  about  the  study  weriB  iijade  by  the  principal  investigator 
and  the  project  manager  at  the  local  meeting  of  these  chapters •    As  a 
result  we  received  ah  Indication  of  interest  from  about  40  outside 
searchers^  36  of  these  eventuaiiy  participated  from  beginning  to  end, 
completing  all  the  tasks*    The  searchers  were  paid  $100  for  their 
time.    Each  of  the  36  searchers: 

•••received  a  description  of  the  study 

• • •was  tested  on  the  three  cog:nitive  tests  described  above 

*^*filled  out  two  questionnaires,  one  dxi  their  profile  (Form  2) 
and  the  other  about  their  search  experience  (Form  3) 

il'^^^^iy^^.i^^^'^^^tio^       procedures  for  searching  in  a 
presentation  and  in  writing  (Procedure  No^  3,  4,  and  5, 
Appendix! ),  including  detailed  instructions  for  using  the 
equipment  and  software  for  log-in  and  downloading 

i>  •received  5  or  6  questions  for  searching  exactly  as  each  was 
received  from  the  user  in  written  form*  together  with  the  user's 
instructions  for  search  constraints 

•• •prepared  a  preliminary  search  strategy  and  recorded  it  on  a 
form  (Form  13) 

••^conducted  the  search  and  recorded  the  results*     (The  whole 
search  was  recorded  on  a  disk  and  on  a  printouts) 

There  were  no  time  limits  placed  on  searchers,  nor  were  any 
requirements  placed  on  the  search  strategies  they  chose  to  prepare* 
This  required  extreme  flexibility  on  the  part  of  the  project  team,  to 
accomodate  to  the  searchers*  schedules • 


6.3  Searching 


Searching  was  done  on  DIAI1G6,  the  largest  and  most  frequently 
used  information  vendor  in  the  world,    DIALOG  supplies  access  for 
searching  to  over  150  databases  •    All  searchers  had  extensive 
experiet^e  on  DIALOG. 

 Each  question  was  searched  on  one  database  only,  selected  by  the 

study  team  on  the  basis  of  closeness  of  fit  between  the  subject  of 
the  question  and  the  subject  coverage  of  the  database^    The  searchers 
were  assigned  questions  which  matched  their  ovm  subject  and  database 
experience,  as  reported,  as  closely  as  was  possible •-    Thus  for 
instance,  searchers  that  Search  medical  databases  most  often  received 
medical  questions.    To  control  for  any  effect  question  order  had  on 
search  results  of  a  searcher,  the  order  in  which  a  question  was 
searched  by  the  five  people  who  searched  it  was  varied. 


Searching  was  done  on  an  IBM-PC  and  on  a  Compaq  microcomputer i 
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(Compaq  is  IBM-PC  cdmpatlBle,  using  the  same  operating  system, 
connnunlcat ion  software,  etci    Thus,  for  all  practical  purposes  these 

were  identical  machines*)  Both  microcomputers  had  a  Hayes  Smart com 

mbdembullt  In  f or  teleconm  the  Smartcbra  software  was 

used  for  communication  and  for  downloading.    Appropriate  cpramands 
Were  pre-programmed i    With  one  stroke  a  aeries  of  instructions  for 
log-in  through  any  of  the  available  carriers  (DIALNET,  TELENET, 
TYtWET,  WipSf )  ras  availab^^^    all  procedural  matters  were  reduced  to 
macros  for  ail  df  the  searchers. 

 Searchers  were  scheduled  according  to  their  time  availability^ 

however,  no  more  than  two  searchers  could  be  accomodated  at  any  one 
time  because  only  the  two  identical  micrcoraputers  were  used. 

6.4  Project  Searches 


As  mentioned,  each  question  was  searched  by  five  outside 
searchers.    In  additibh,  there  were  four  searches  done  ±n-house  by 
the  project  staff.    These  additional  searches  were  labeled  project 
searches,  and  the  staff  searchers  as  project  searchers.    Thus,  nine 
searches  were  done  for  each  question:  five  by  outside  searchers  and 
four  by  project  searchers. 

The  four  project  searches  were  designed  to  study  various  types 
of  searches  resulting  from  different  sources  for  the  search  strategy: 

Project  Search  1:  done  on  the  basis  of  only  the  taped  interview 

with  the  user  describing  the  problem 
underlying  the  question  and  the  intended  use 

Project  Search  2:  done  on  the  basis  of  the  taped  interview  plus 

the  written  question  statement 

Project  Search  3;  done  on  the  basis  of  only  the  written  question 

statement  using  only  the  terms  in  the  written 
statement  and  lib  elabpratibhf  (this  approach 
is  like  an  autbmatic  language  processor 
preparing  the  question  for  searching) 

Prx>jeet  Search^:  done  bn  the  basis  of  the  written  question 

statement  plus  elaboration  using  the 
apprbpriate  thesaurus 

The  project  searches  were  done  by  three  project  searchers.  As 
mentioned,  the  three  project  searchers  conducted  the  taped  interview 
id.th  the  users.  However,  the  project  searches  were  arranged  so  that 
they  were  done  by  the  pro ject  searcher  who  did  not  do  the  interviews 
Thus,  each  project  searcher  heard  the  taped  inter^rlew  for  the  first 
time  when  the  first  project  search  was  done.  Procedures  for  project 
searches  are  reproduced  under  Procedure  No.  8  in  Appendtx  ls 

As  mentioned,  there  were  36  outside  searchers,  three  project 
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searchers^  and  aii  analyst  use^  to  test  of  the  consistency  of 
determinations  of  question  strucutre  and  classif icatidti.  Aitbgether 
there  were  40  searchers  involved i    Each  question  was  searched  nine 
times  (five  outside  se^  project  searches^  and  there 

were  40  questlonsi  thus  altogether  there  were  360  searches  performed. 
Looking  at  the  number  of  searches  another  way,  40  questions  received 
U5  five  outside  searches  for  a  total  of  5  X  40  *  200  outside 
searches;  and  (±±)  four  project  searches  for  a  total  of  4  X  40  »  160 
project  searches.    The  200  outside  searches  were  divided  among  36 
outside  searchers  so  that  each  person  did  five  or  six  questions. 

6.5  Qaestion  Stguc^ure^and  Classification 


This  part  of  the  overall  study  had  two  objectives: 

1.  For_ questions  in  general,  to  test  the  consistency  of 
application  of  a  postulated  structure  and  categorical 
classification 

2.  For  the  40  questions  which  were  part  of  the  study  proper,  to 
provide  classification  into  categories  and  to  use  the 
classifications  as  one  of  the  variables 

The  consistency  of  assigning  structure  and  classification  was 
tested  by  having  tw  judges  assign    Independently  from  each  other * 
the  appropriate  function  designations  (lead-in,  subject,  query, 
modifiers,  constraints)  and  categories  (level  of  clarity, 
specification,  complexity  and  presupposition).    Two  sets  of  questions 
were  used  to  test  this  consistency: 

li  The  40  questions  used  in  the  study  proper 

2.  An  additional  178  questions  collected  during  the  preceeding 
study  (1981-83) 

 The  two  groups  of  questions  and  their  classifications  were 

treated  separately,  because  the  additional  178  questions  had  nothing 
to  do  with  the  study  proper  and  were  used  only  as  additional 
supporting  (or  refuting)  data  for  the  test  of  question 
cl^issification. 


(5.6  Handling  gf  Data 


A  set  of  procedures  was  designed  and  programs  were  written  for 
handling  the  variety  of  data  obtained  during  searching^  The 
flowchart  of  these  procedures  is  presented  in  Appendix  1 .  Briefly, 
the  following  was  involved: 

...The  end  result  of  each  search  was  a  list  of  accession  numbers 
of  items  retrieved 
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•  ••the  accession  nombers  for  retrieved  Items  from  niiie  searches 

merged  and  a  union  set  created 
(duplicates  were  eliminated) 

•••If  the  union  of  retrieved  items  exceeded  150|  only  the  first 
150  items  were  designated  to  be  sent  to  the  user  for  evaluation* 
This  was  done  to  make  evaluation  manageable  for  the  user  who  if 
presented  with  aii  bverwheljiiing  output  might  have  considered  the 
task  unreasonable,  and  rightly  so,  and  may  not  have  finished i 
Since  all  searchers  had  ah  equal  chance  to  contribute  to  the 
retrieved  set,  including. the  first  150  items,  there  was  no  bias 
toward  aiiy  searchers.    DIALOG  databases  are  organized  on  a  last 
in/first  but  principle^  thus  the  first  150  itens  represent  the 
most  recent  additions  to  the  databases  and  the  literature • 

...The  full  record  of  each  item  ret rie^^ed  in  the  union  set  of 
150  was  downloaded  from  DIALOG  onto  floppy  disk  and  then 
printed.    In  most  cases  this  represented  bibliographic  data, 
indexing  and/or  classification  and  an  abstract. 

...To  each  item  retrieved  a  line  for  evaluation  was  added: 

-   Relevant  Partially  Relevant   Ndnre levant 

The  printout  and  a  carbon  copy  were  sent  to  the  mer  for 
evaluation^  The  user  returned  the  original  with  his/her 
evaluation. 

were  recorded  with  the  accession  number  of 
trieved  for  a  question  to  serve  as  the  benchmark 
file  against  which  the  output  for  each  search  was  compared. 

••L^^^™^^®  P'^'^tocol  of  each  search,  data  vera  obtained  about 
terms,  commands,  and  cycles  used  for  comparison  with  each  other; 
in  addition  the  preparation  and  connect  times  were  recorded  for 
each  searcher. 

. . .A  large  inaster  data  file  was  created  cdntaihihg  data  for  each 
of  the  variables »  i_.e.>  user  evaluation  for  each  item  retrieved 
by  each  search^  utility  measures  for  each  question,  efficiency 
measures  for  each  search^  context  indicators,  searcher 
characteristics!  question  classification  indicators,  etci 

...This  master  file  was  used  as  a  basis  for  all  statistical 
analysis  by  creating  subfiles  for  each  set  of  interacting 
variables  (i.e. ^  analysis  questions  asked)  and  then  analyzing 
those  using  standard  statistical  analysis  methods. 

While. some  20  programs  were  written  to  handle  the  data,  at  each 
8tep_ manual  verification  and  editing  was  done  to  eliminate  errors  and 
resolve  ambiguities. 
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1 .  SUMMARY  LIST  AND  CODE  BOOK  FOR  VARIABLES 


The  90  variables  are  presented  In  Table  7-1 >    The  list  of 
variables  is  organized  by  categories  of  entities  as  described  in  the 
preceedlhg  sections*    The  following  is  given  for  each  variable: 

!•  A  code  which  was  used  in  statistical  analysis 
2»  Name 

3.  Short  description  of  the  measure  or  quantity  defined 
km  Source  of  the  data 

 3  code  book  in  the  study  and  is  a  major  tool 

in  statistical  analysis  and  interpretation  of  the  data •    It  is 
reproduced  here  in  its  entirety  to  provide  a  quick  reference  to 
variable  descriptions.    In  the  part  of  this  report  on  the  statistical 
analysis,  there  are  repeated  references  to  this  code  book* 
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TABLE  7-1 :  DESeRIPTiGN  OF  VARIABLES  FOR  WHICH 

QUANTITATIVE  DATA  WERE  COLLECTED  AND  THE 
CODE  BOOK  FOR  STATISTICAL  ANALYSIS 


ENTITY 


CODE 


NA^ffi  AND  DESCRIPTION  OF  VARIABLE 


RETRIEVED  1.  EVAt 
ITEM 


Eyaloatiang  of  Retrieved  It^ms  -  A  merged  set  of 
the  full  records  of  items  retrieved  by  nine 
searches  for  each  of  the  40  questions  was  sent  to 
the  4b  users  for  evaluation.    Each  item  was 
evaluated  as  either  "relevant",  "partially 
relevant",  or  "not  relevant". 


SEARCH 


2.  SVA 

3.  SVB 

4.  SVC 

5.  SVD 

6.  SVE 

7.  SVF 

8.  SVG 

9.  SVH 

10.  SVI 

11.  SVJ 

12.  svk 


Nifflber  of  Relevant  Items  Retrieved  by  a  Search  - 
Number  of  items  retrieved  by  an  individual  search 
of  a  question  and  evaluated  by  the  user  as 
relevant  _ 

Number  of  Partially  Relevant  Items  Retries 
Search  -  Number  of  items  retrieved  by  an 
individual  search  of  a  question  and  evaluated  by 
the  user  as  partially  relevant 

Number  of  Not  Relevant  ItCTS-Jletrleved-by  a  Search 

Number  of  items  retrieved  by  an  individual 
search  of  a  question  and  evaluated  by  the  user  as 
not  relevant 

Total  Ntmber  of  Itema-Jtetrieved  by  a  Search  and 


Evaluated  Jxy 


-  Total  number  of  items 


retrieved  by  an  individual  search  of  a  question 

^^^f'^^i^^^^^.^y  user 

Items  ^etr±eved-by-a  Search  but^^tet^valuated  by 
the  User  -  Number  of  items  retrieved  by  the 
individual  search  of  a  question  but  not  evaluated 
by  the  user 

Total  Ntmber  of^ Items  Retrieved  by  a  Search  of  a 
Question 

Search  Recall^  -  The  percentage  of  relevant  items 
retrieved  by  an  individual  search  of  a  question  in 
comparison  to  the  total  number,  of  relevant  items 
retrieved  by  all  searches  of  the  same  (question 
Search  Precisibn  -  The  percentage  of  relevant 
items  retrieved  by  ah  individual  search  of  a 
question  iii  comparison  to  the  total  number  of 
items  the  iearch  retrieved  which  were  evaluated 
Number  of  Commands  Used  in  a  Search  -  Total  number 
of  Dialog  "select"^  "expand",  "combine"  and  "type" 
commands  used  by  a  searcher  searching  a  question 
Number  of  Command  Cycles  Used  in  a  Search  -  Total 
number  of  sequences  of  "select"/"combine"/"type" 
used  by  a  searcher  searching  a  question 
Number  of  Search  Terns  Used  in  a  Search  -  Search 
terms  "selected"  by  a  searcher  searching  a 
question.    Search  terms  entered  as  a  set  number 
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after  an  "expand"  command  were  riot  counted. 

13.  SVL  dniln^-Connect  Time  Used  for  a  Search  -  The  time 

spent  online  by  a  searcher  searching  a  question, 
counted  iri  fractloris  of  an  hour 

14.  SVM  ggeparafeloa^Tlme  Used  far  a  Search  -  The  time 

spent  by  a  searcher  preparing  to  search  a 
guest Ipri,  counted  In  fractions  of  an  hour 

15.  SVN  To^l  Time  Used  for  a  Search  -  The  time  spent  by  a 

searcher  orillrie  added  to  the  time  spent  offline 
preparirig  to  search  a  question,  counted  in 
fractions  of  an  hour 

QUESTION        16.  QVA  Total  Number  of  Relevant  Items  Retrieved  ^or  ^ 

Question  "  Of  the  total  number  of  Items  retrieved 
by  all  searches  of  the  question  (the  merged  set), 
  those  evaluated  as  relevant 

17.  QVB  Total  Number  of  Partially  Relevant  tt^s -Retrieved 

for  a  Question  -  Of  the  total  number  of  items 
retrieved  by  ail  searches  of  the  question,  those 
evaluated  as  partially  relevant 

18.  QVe  Total  Jlamber  of  Not^levant-  Items  Re^gf.eved^or^ 

Question  -  Of  the  total  number  of  items  retrieved 
by  ail  searches  of  the  question,  those  evaluated 
as  not  relevant 

19.  QVD  fota£^umber  of  Evaluated  I  tems^  Retrieved  fog  a 

Question  -  Of  the  total  number  of  items  retrieved 
by  all  searches  of  the  questiori,  those  the  user 
evaluated 

26.  QVE  Total  Number  of  Not  Evaluated  Items  Retrieved  for 

a^  Question  -  Of  the  total  number  of  items  ~~ 
retrieved  by  all  searchers  of  a  questibrii  those  * 
items  the  user  did  riot  evaluate 

21.  QVF  Total  Number  of  Items  Retrieved  for  a  Question  - 

The  number  of  items  iri  the  uriibri  or  merged  set  of 
all  items  retrieved  by  all  searches  of  the 
guestipri_(elimlriating  duplicates 5 

22.  QVH  Overall  Precision  -  Percentage  of  relevant  or 

partially  relevarit_  items  for  a  question 

23.  UVA  Time  Sperit  by  the  User  Evaluating  the  Items  -  The 

user  was  asked  bri  a  questionnaire  (Form  12),  "How 
much  time  did  ybu  spend  resviewlhg  these 
abstracts?"    The  questionnaire  (Form  12)  was  sent 
albrig  with  the  retrieved  items  he  or  she  received. 
Time  spent  evaluating  was  counted  as  hours  and 
  fractions  of  an  hour. 

24.  UVB  Dollar  Value  Assigned  -  The  user  wa^  also  asked  on 

the  questionnaire  (Form  12) ^  "in  an  overall  sense, 
if  ybu  were  asked  to  assign  a  dollar  value  to  the 
usefulness  of  this  entire  set  of  abstracts  to  you, 
what  would  that  dollar  value  be?"  (fill  in  the 
blank)   

25.  UVC  Worth  Assigned  -  The  user  was  asked  on  the 

questionnaire  (Form  12),  "Could  you  rate  your 
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26.  UVD 


27.  UVE 

28.  PREBRD 

29.  APPLN 

30.  LANG  : 

31.  YEARGEN 

32.  YEARSPl 

33.  YEARSP2 

34.  SUGDBi 

35.  SUGDB2 

36.  SUGDB3 


participation  ±n  this  project  and  the  irifdrmatibn 
that  has  resulted  jfrom  it  as;  Worth  much  more  than 
^55,  worth  somewhat  mora  than  (4),  worth  about  as 
much  as  (3)^  worth  less  than  (2)  the  time  it  has 
taken,  or  practically  worthless  (1)?" 
Problem  ReBolutioo.  -  The  user  was  asked  on  the 
questionnaire  (Form  12),  "On  a  scale  of  1  to  5^ 
what  contribution  has  this  information  made  to  the 

<5f  the  problem  that  motivated  your 
question?  Substantial  contribution  (5)  . /.  nothing 
contributed  (1)"  ^ 
Satls^actlw  -  The  user  was  asked  ph  the 
questionnaire  (Form  12),  "On  a  scale  of  1  to  5^ 
how  satisfied  were  you  with  the  results  of  the 
search?  Satisfied  (5)  ...  dissatisfied  (1)" 
User  Requested  a  Broad  or  a  Preciise  Search  -  At 
the  time  the  user  was  given  the  opportunity  to 
submit  hiiorher  written  question^  he  was  asked 
(bh  Form  55^  "Do  you  want  a  precise  search  (3), 
pr^a  broad  search  (45? 

Application^of  Research  -  Also  on  Form  5,  the  user 
was  asked >  "Type  of  application  of  this  research: 
undergraduate  study  (5),  graduate  study  (6), 
faculty  research  (7)^  industrial  research  (8), 
general  (9)^  or  other  (105." 

LimitRetrieval  by  Language  of  Publicatton  -  On 
Form  5  the  user  was  asked »  "Do  you  want  to 
restrict  the  language  of  publication  of  the 
articles  retrieved?  English  only  (11),  or  any 
language  (12)" 

Limit  Ret reival  by- Year  of  Publication  -  On  Form  5 
the  user  was  asked,  "Do  you  want  to' restrict  the 
years  of  publication  of  the  articles  retrieved? 
last  5  years  (13)^  or  no  limits  (14)" 
Limit  Retrieval  by  Year  Qf-PublAcation^^^m  ...  - 
On  Form  5  the  user  had  an  option  to  specify  a  year 
of  the  oldest  articles  desired  (fill  in  the  blank) 
LimiiuRetrieyal  by  Year-of-Pufeiicatibn  to  . . .  -  On 
Form  5  the  user  had  an  option  to  specify  a  year  of 
the  most  recent  articles  desired  (fill  in  the 
blank) 

Database  Suggested  by  User  (first  choice)  -  On 
Form  5  the  user  was  giveii  an  optiph.  to  suggest  a 
Dialog  database  (by  hame\  i.e.,  DIALOG  file 
number)  fbr  the  search  of  his  or  her  question 
(fill  in  the  blank) 

Database  Suggested  by  User  (second  choice)  (fill 
in  the  blank) 

Database  Suggested  by  User  (third  choice)  (fill  in 
the  blank) 


SEARCHER       37.  FREQDIAL    Frequency  of  Searcher *s  Use  of  Dialog  -  Before 

participating  in  the  searching  each  searcher  was 

38 
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38.  DBASE 1 


39,  DBASE2 
4e,  DBASE3 

41.  DBASEA 

42.  DBASES 

43.  DBASE6 

44.  DBASE? 

45.  FREQl 


46.  FREQ2 

47.  F^Q3 

48.  F^Q4 

49.  FREQ5 

50.  FREQ6 

51.  FREQ7 


asked  on  a  qaestionnaire  (Form  3),  '^ow  often  do 
70a  search  Dialog?  Daily  (5),  Twice  a  week_C4)j 
Once  a  week  (3),  Twice  a  month  (2),  Less  (1)" 
Dialog- Database  Used  Most^^ffeea  -  Oh  the 
qaestionnaire  (Form  3)  the  searcher, was  asked  to 
select,  from  a  subject  categorized  list  of  85 
Dialog  databases,  the  database  heorshe  used  most 
oiften.    Answers  were  recorded  as  DIALOG  file 
numbers. 

 sed Second  Host  Often 

 abase-gsed  Third  Most  Often 

Diateg-^a^baae  Used  Fourth  Most  Often 
Dialog  Database  Used  Fifth  Most  Otten 
Dialog  Database  Used  Sixth  Most  Often 
Dialog  Database  Used  Seventh  Most  Often 
Frequency  of  Use  of  Dialog  Database  Used  Most 
Often  -  Oh  the  questionnaire  (Fotm  3)  the  searcher 
was  asked  how  often  he  or  she  used  the  Dialog 
database  used  most  often:  Daily  (5),  Twice  a  week 
14),  Ohce  a  week  (3),  Twice  a  month  (2),  Less  (1) 
Frequency  of  Use  of  Dialog  Database  Used  Second 
Most  Often 

Frequency  of  Use  of  Dialog  Database  Used  Third 
Most  Often 

Frequency  of  Use  of  Dialbg  database  Used  Fourth 
Most  Often 

Frequency  of  Use  of  Dtalog^^atabase  Used  Fifth 
Most  Often 

Frequency  o£  Use  ^ dialog  Database  Used^^lxthr 
Most  Often 

Frequency  of  Use  of  Dialog  Database  Used  Seventh 
Most  Often 


QUESTION 

AND 

USER 


52.  PROBDEFl 


53.  iNTENTl 


54.  PROBKNOl 


definition- by  User  -  During  ah  ihteryiew 
the      user  was  asked  (the  response  was  recorded  oh 
Form  7),  "In  your  opinion  and  on  a  scale  from  1  to 
5,  would  you  describe  yoiir  problem  as  weakly 
defined  or  clearly  defined,  with  1  being  weakly 
defined  and  5  being  clearly  defined?" 
Inteat-by  User  -  Dtirihg  the  interview  the_user  was 
asked  (the  response  was  recorded  oh  Form  7)^  "Oh  a 
scale  from  1  to  5,  would  you  say  that  your  use  of 
this  ihf brmatibh  will  be  opeh  to  many  avenues^  or, 
for  a  specifically  defined  purpose,  with  1  being 
open  to  mahy  avenues  and  5  being  a  specifically 
defined  purpose?" 

Prbblem'"Public  Knowledge  by  User  -  On  a 
questionnaire  (Form  8)  presented  to  the  user  at 
the  interview^  he  or  she  was  asked^  "On  a  scale 
from  1  to  5^  how  would  you  rank  the  prbbabtiity 
that  information  about  the  problem  which  motivated 
this  research  question  may  be  found  in  the 
literature?  Highly  improbable  that  it  exists  (1) 
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55.  INTKNOI 


Highly  probable  that  it  exists  (5)" 
Internal  Knowl^jjge-of^Usar^  -  On  the  quest Ibhiialre 
(Form  8)  the  user  was  asked,  "On  a  scale, from  1  to 
5,  how  would  you  rank  the  amount  of  knowledge  you 
possess  in  relation  to  the  problem. which  motivated 
this  request:  Little,  personal  knowledjge  (1)  ... 
Considerable  personal  knowledge  (5)" 


QUESTION 
AND 

PROJECT 
SEARCHER 


56.  PR0BDEF2 


57.  INTENT2 


groblein  Defiriitibn  by  Searcher  ^  The  project 
searcher  was  asked  after  listening  to  the  tape 
recorded  interview  with  the  user  (the  response  was 
entered  on  Form  9)^  "In  your  opinion  and  on  a 
scale  from  1  to  5^  would  you  describe  the  user^s 
problem  as  discernible  from  the  interview  as 
weakly  defined  or  clearly  defined?" 
Intent  by  Searcher      The  project  searcher  to 
asked  on  Form  9  after  listening  to  the  taped 
interview^  "On  a  scale  from  1  to  5,  would  you  say 
that  the  use  of  this  information  by  this  user  will 
be  open  to  many  avenues  or  for  a  specifically 
defined  purpose?" 


QUESTION 

AND 

ALL 

SEARClffiRS 


58.  PR0BKN02 


59.  iNTKS02 


Problem'^Pubiic  Xnow±edge-J?y— Searcher  -  The  project 
searcher  was  asked  (Form  9)  and  the  searcher  was 
asked  (Form  16)  at  the  time  he  or  she  prepared  to 
search  the  question  online,  "How  would  you  rank 
the  probability  that  information  about  the  problem 
which  motivated  this  research  question  may  be 
found  in  the  literature?  Highly  improbable  thatit 
exists  (i)  ...  Highly  probable  that  it  exists  (5)" 
internal  kaowledge  ^  Searcher  -  The  pro: 


searcher  was  asked  (Form  9)  arid  the  searcher  was 
asked  (Form  10),  '•How  wbuld  you  rank  the  amount  of 
knowledge  you  possess  iri  relatipri  to  the  problem 
which  motivated  this  request?  Little  personal 
knowledge  (1)  ...  Considerable  personal  knowledge 
(5)" 


^estion.  Analysis  -  The  following  variables,  numbered  60.  through  81., 
relate  to  the  analysis  of  the  40  research  questions  done  independently  by 
two  different  judges.    Values  for  each  variable  for  each  question  were 
determined  by  the  judges  arid  recorded  on  a  copy  of  Form  Hi 

60.  CATOGORY    Question  Classification  (Judge  1);  Domain! 

Diallndex  Categories  -  Using  a  copy  of  the 
categories  of  databases  published  by  Dialog,  the 
judge  determined  the  subject  categories 
appropriate  for  searching  the  questions 
Question  Classification  (Judge  tX^J^arityt^ 
Semantics      Was  the  meaning  of  the  terms  used  in 
the  written  question  clear?  Unclear (0) .. .Clear (5) 

40 


QUESTION 
CLASSI- 
FICATION 
(JUDGE  1) 


61.  CLTYSEM 


ERLC= 


61 


62.  CLTYSYN 

63.  CLTYAVG 

64.  SPECQURY 

65.  SPEGSUBJ 

66.  SPECSffiAN 


67.  SERCONC 

68.  CONSTRAN 

69.  TRANSSER 

70.  FINAL 


Question  Classification  (Jad&e- 1 ?;  Clarity^  Syntax 
-  Was  the  rielation  between  the  terms  of  the 
written  question  clear?  Unclear  (0)  ...  Clear  (5). 
Question  Classification  (Judee-lH-Clar4ty-Score  - 
determined  by  adding  the  clarity:  semantic  score 
to  the  clarity:  syntax  score  and  dividing  by  2. 
Ques  t  ion  Glasslflcatlon  X Judge  1     Speclrficl  ty : 
Query^-  We  re  the  query  terms  (query  terms  were 
determined  using  a  special  definition  developed 
for^the  question  classification  process)  meta 
termsor  object  terms?  Meta  (0)  ...  Object  (5) 
Question  Ciaaaiflcatlon^^Judge  1);  Specificity; 
Subject  -  Were  the  subject  terms  (subject  terms 
were  determined  using  a  special  defihitibh 
developed  for  the  question  classification  process) 
meta  terms  or  object  terms?  Meta  (0). ..  Object  (5) 
4jxiestio&  Classification  (Judge  1);  Specificity 
Score  -  determined  by  adding  the  specificity: 
query  score  and  the  specificity:  subject  score  and 
dividing  by  2. 

^uestl^  Classification  (Judge  1):  Number  of 
Search  Concepts  -  determined  as  a  total  ntnuber  of 
concepts  involved  in  the  question 
Qyestibh  Classification  (Judge  1);  Ntmber  of 
Constraints  ^  f or  example  the  years  of 
publications  desired 
Question  Classification  (Judge  i): 
Presuppbsitibns;  Number  trans ferable^g  Search 
Prbcess  -  fchbwledge  presupposed  by  the  question 
Questibn  Classification  (Judge  lX:-^nai  Summary 
scbre  -  no  value  was  actually  attached  as  a  final 
score 


Variables  71.  through  81.  are  a  repeat  of  the  question  analysis  variables 
just  described  except  they  relate  to  the  second  judge. 


QUESTION 
CLASSI-_ 
FICATION 
(JUDGE  2) 


71.  CATEG2       Question  Classiflx 

Dlallndex  Gategorlea 

72.  CLTYSEM2    Question  iS3 


Semantics 
73.  CLTYSYN2  Question^ 


Judge  2):  Clarity: 


74.  CLTYAVG2    Question  Claself icatlon 


city:  Syntax 


M  Clarify  Scbre 


75.  SPEeQRY2    Question  Classification  (Judg^ 2^;  Specificity: 

Qoe^cyL 

76.  SPE6SUB2    Question  Classif Icatlon  (Ju<ige  2):  Specificity: 

Subject 

77.  SPEQffiN2    Question  Clasalficatlonr  (Judge  2):  Specificity 

Score 

78.  SERebNC2    Question  Classlf^catlc     (Judge  2):  Number  of 

£4&pt# 


79 i  CbNSTR2 


QuestloH^^assif icatibh  (Judge  2):  Number  of 
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80.  TRANSER2    Quest^oa^^lasslf Icatlon  iJudge  2); 

Presuppositions;  Number  Transferable  to  Search 
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Process 

81.  FiNAt2       Qaestlon  Ciasslf tca^Aon  (Judge  2);  Final  Summary 


The  following  variables  82*  through  90.  were  all  derived  from  the  three 
cognitive  tests  administered  to  the  searchers. 

Semote  Associates— Test^ 

82.  RAT  Rcimote^ssociates  Test  Score  (Searcher)  -  The  test 
was  designed  to  measure  verbal  facility  and  verbal 
creativity.    The  test  allowed  20  minutes  for  each 
ihdlvrdtial  to  answer  a  possible  30  questions. 
Each  question. asked  the  person  to  supply  a  fourth 
word  which  related  to  each  of  three  words  which 
were  glven^    Example:  cookies^  sixteen,  heart, 
(sweet).    The  hljghest  score  possible  was  30. 

Employee  Aptitude. Survey 

83.  EAS  Employee  Aptitude  Survey  Score  (Searcher)  -  this 
was  a  test  of  symbolic  reasonings    The  person  had 
5  minutes  to  complete  a  possible  30  questions. 
Example:    A  >  BjC  e,  therefore,  A">  C.  There  were 
three  possible  answers:  True^  False,  or  ?  The 
highest  possible  score  was  30. 

Learning  Style  inventory  

The  test  included  12  questions  designed  to  allow  a  person  to 
describe  his  individual  learning  style,    the  Learning  Style 
Inventory  was  scored  in  parts  as  follow  depending  upon  whether 
the  person learning  style  was  characterized  by  Concrete 
Experience  (feeling).  Reflective  Observatldn  (watching). 
Abstract  Conceptualization  (thinking),  or  Active  Experience 
(doing): 


SEARCHER 
CHARACTER- 

iSTies 


CE  Learning  Style  Inventory  (Searcher):  Cbhcrete 

Experience  Score  -  ranging  from  12  to_48_. 

85.  RG  Learning  Style  Inventory  (Searcher):  Reflective 

Ot^servatl^n.  Score  -  ranging  from  12  to  48. 

86.  AC  Learniti^^t^le  Inventory  (Searcher);  Abstract 

CoaceptAializatlbh  Score  -  ranging  from  12  to  48* 

87.  AE  Learning  Style  Inventory  (Searcher):  Active 

Experimentation  Score  -  ranging  from  12  to  48. 
8i3.  ACCE  Learning  Style  Inventory  (Searcher):  Abstract 

Cohceptuallzatlbn/Cdncrete  Experience  Score  (AC 
minus  CE)—  ranging  from  +36  to  -36* 

89.  AERO  Learning  Style  Inventory  (Searcher):  Active 

Experlmentatlbh/Reflective  Observation  Score  (AE 
mlhuis  RO)  -  ranging  from  +36  to  -36  i 

90.  LSI  Learning  Style  Inventory  (Searcher):- j*inal  Score 

(this  score  was  not  used  as  a  variable  for  the 
present  study). 
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PART  I'jCI.  BASie  DATA 


8«  BACKGROUND  DATA  ON  U§ERS  AND  QUESTIONS 


8«1  Users 


 The  study  involved  40  users,  each  submitting  one  question  for 

searching.    This  section  contains  STiinmiary  data  oh  the  40  users. 

 Table  8^1  breaks  down  the  users  by  occupation.    As  can  be  seen, 

about  48%  of  the  users  were  faculty,  37%  were  graduate  students,  and 
15%  were  from  industry.    The  table  also  lists  the  questions  (by 
question  number)  associated  with  each  group.    Full  text  of  each 
questions  is  contained  in  Appendix  A  and  a  short  title  is  listed  in 
Table  8^. 

Another  way  to  characterize  the  users  is  to  list  the  type  of 
work  the  user  was  doing  related  to  the  question  submitted  as 
presented  in  Table.  8-2  ,    As  can  be  seen,  35%  of  their  work  was  done 
as  faculty  research,  A2Z  as  graduate  study,  13%  as  industrial 
research,  and  10%  was  general  or  other. 


8.2  Questions 


Short  summaries  of  the  40  questions  are  provided  in  Table  8^S. 
The  table  also  indicates  the  question  number  assigned  in  the  project 
and  the  I^IALOG  file  nraaber  on  which  the  question  was  searched.  Full 
text  of  questions  is  in  Appandix-A. 

 liters  had  the  choice  of  requesting  various  restrictions  which 

coaid  be  applied  to  the  search  of  their  question.    Table  8-4  provides 
a  somnary  of  user  requests  pertaining  to  level,  language^  and  years 
to  be  searched,  and  DIALOG  databases  suggested. 


 ^  seen,  for  90%  of  the  questions^  broad  searches  were 

requested;  63%  requested  articles  in  English  language  only;  63% 
placed  no  limits  on  year  of  publication  retrieved;  and  80%  suggested 
no  specific  DIALOG  file. 


8.3  Context 


Each  user  assigned  a  value  for  the  context  measures  pertaining 

to: 

1.  How  well  was  their  problem  defined,  from  1  (poorly  defined) 
to  5  (well  defined) 
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2.  Was  their  tntent  well  formulated,  from  1  (open  to  many 
avenues)  to  5  (narrowly  defined) 

3.  What  was  the  probability  that  fmbiic  kriowiedge  existed  on  the 
subject  of  their  request  from  1  (low  level— highly  improbable 
that  it  exists)  to  5  (high  level—it  exists) 

4.  What  was  their  ia^ernal  knowledge  on  the  problem  giving  rise 
to  the  quej^tion,  from  1_  (little  personal  knowledge)  to  5 
(considerable  personal  knowledge) 

Values  assigned  to  each  questionare  given  in  Appendix  Aj 
together  with  the  written  statement  of  the  question.    Table  8-5 
provides  data  on  the  cumulative  scores  for  each  of  the  five  values 
over  40  questions.    For  example,  it  shows  that  10  users  indicated 
five  (well  defined)  on  their  problem  definitions i    It  also  shows: 

•••58%  of  the  users  considered  their  problem  quite  well  defined 
(top  two  scores) 

• • ^45%  thought  that  their  intended  use  could  be  close  to  "open 
to  many  avenues"  (lowest  two  scores) 

•••60%  indicated  there  was  close  to  certainty  that  information 
requested  can  be  found  in  public  knowledge  (top  two  scores) 

•••45%  considered  themselves  quite  knowledgable  about  the 
problem  at  hand  (top  two  scores) 

Project  searchers  were  also  asked  to  assign  context  values  for 
each  question  they  searched.     (As  mentioned^  pro ject  searchers  did 
four  searches  on  the  basis  of:  1.  Taped  problem  statement  :f  users; 
?i_?^P?^  P.^^^i™  statement  plu&  written  question  statement;  3. _Terms 

^J^^^^^  question  statement  only  (no  elabpratibn)i  4.  Terms 
from  the  written  question  statement  plus  elaboration  by  thesaurus.) 

The  project  searchers  listened  to  the  tape  of  the  user 's  problem 
and  intent  and  formed  their  own  opinion  of  the  context.    In  addition, 
they  indicated  their  own  Internal  Knowledge  about  the  problem  at 
hand.    The  summary  of  this  data  is  presented  in  Table  8-6  which 
shows : 

...the  problem  underlying  50%  of  the  questions  was  considered  by 
the  project  searcher  to  be  quite  well  defined,  in  contrast  to 
58%  by  the  users 

.. .intent  was.considered  by  project  searchers  as  "open  to  many 
avenues"  for  50%  of  the  questions,  in  contrast  to  45%  indicated 
so  by  users 

...for  30%  of  the  questions  project  searchers  indicated  that 
there  was  close  to  certainty  that  information  requested  could  be 
found  in  puMic  knowledge,  in  contrast  to  60%  indicated  so  by 
the  users 
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•..fbr_85%  of  the  questions ^  project  searchers  considered 
themselves  having  quite  low  knowledge  about  the  problem  at  hand, 
whereas  45%  of  the  users  believed  they  possessed  high  knowledge 
and  10%  low  knowledge 

Thus I  there  were  differences,  to  varying  degrees,  between 
project  searchers  and  users  in  their  respective  assessments  of 
context  questions.    The  differences  were  not  large  except  in  internal 
knowledge  and  the  existence  of  public  knowledges    Project  searcher 
admitted  to  considerably  lower  internal  knowledge  than  did  the  users 
about  the  problem  at  hands    But,  that  should  be  expected.  User 
estimation  of  the  existence  of  public  knowledge  was  much  greater  than 
the  estimation  made  by  project  searchers. 

  Th  ;  outside  searchers  also  were  asked  to  complete  two  context 

measures:    public  knowledge  (what  is  the  probability  of  finding 
^?^9^^?^9^_9^  ?^®_r6quest)  and  internal  knowledge  (how  much,  do  you 
know  about  the  question  at  hand).    They  could  hot  judge  problem 
defiuitions  and  the  Intent  scales  becasue  they  did  not  hear  the  taped 
problem  and  intent  statement's.     (The  36  outside  searchers  searched  on 
the        '      f  rrritten  questions.)    Each  question  was  searched  by  5 
sear'  40  questions  x  5  searches  for  a  total  of  200 

seasr  .cv  '.;he  outside  searchers  provided  200  context  scores, 

^^etai.  r'.xt  &core8  for  each  searcher  are  shown  in  Appendix  A. 

i?!;:  ;  provider  ^  suy^vaary  .aliTe  for  the  two  context  measures 

(public  .  iidwledge  6ir.t2  internal  knowlc  ige)  as  scored  by  36  outside 
searcher.'^:  :^hile  searching  the  40  questions.    As  can  be  seen: 

..« in  about  58%  of  the  scores  (for  those  who  scored),  the 
outside  searchers  believed  there  was  close  to  certainty  that  the 
information  requested  could  be  found  in  public  knowledge,  in 
contrast  to  60%  of  users  for  their  questions  and  30%  of  the 
project  searchers 

...in  about  60%  of  the  cases ^  outside  searchers  considered 
themselves  as  having  quite  low  knowledge  about  the  problem  at 
hand.    In  contrast,  for  85%  of  the  questions  project  searchers 
indicated  low  knowledge  and  10%  of  the  users  indicated  low 
knowledge  while  45%  of  the  users  indicated  high  knowledge. 

The  measure  on  public  knowledge  where  users,  project  searchers, 
and  outside  searchers  gave  their  own  estimate  of  the  availability  of 
iiifonnation  in  public  knowledge,  showed  the  greatest  disparity. 
?^???_?^^?8ht  almost  the  same  as  outside  searchers,  however,  both 
differed  significantly  from  project  searchers. 
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Table  8-1 


OeeUPATiON  GF  USERS  SUBMITTING  QUESTIONS  FOR  SEARCHING  (N  users  »  40) 


Niimber  of 
Questions 

Gc  dpatlon  Submitted  Question  Number 


Faculty 

19 

Q005; 
Q017; 
Q026; 
Q031; 

QS09; 
Qdi9; 
Q027; 
Q032; 

Qdll; 
Qd20; 
Qd28; 
Qd34; 

Qdi3; 
Qd21; 
Q029; 
Q036 

Qdl6; 
Qd24; 
Qd3d; 

Graduate  Students 

15 

QOOl; 
QO08; 
Q018; 

QO02; 
QdlO; 
Q022; 

Q0O3; 
Qdl2; 
Qd25; 

Qdd4; 
Qdl4; 
Qd33; 

Qdd7; 
Qdl5; 
Q035 

Industrial  Affiliation 

6 

QO06; 
Qd4d 

Q023; 

Qd37; 

Qd38; 

Qd39; 

TOTAL 

40 

Table  A-^ 


TYPE  OF  WORK  FOR  WHICH  INFORMATION  WAS  REQUESTED 
(N  questions  »  40) 


Number  of 

Type  of  Work  Questions 

Faculty  Research  14 

Graduate  Study  17 

Industrial  Research  5 

General  1 

Other  3 
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SUMMARY  OF  THE  SU  OF  EACH  OF  THE  40  QUESTIONS  TOGETHER  WITH  THE 

NUMBER  OF  THE  DIALOG  FILE  ON  WHICH  THE  QUESTION  WAS  SEARCHED 


Question 
Number 


QOQl  The  relationship  and  communication  processes  between  middle 

aged  children  and  their  parents     (searched  in  DIALOG 
Fiiell) 

Design,  stxructuire,  and  organization  including  overall 
integration  of  the  acute  care  nursing  department  in  the 
hospital    (searched  in  DIALOG  File218) 

Stereotypes  which  affect  the  diagnosis  of  child  abuse  by 
health  care  providers     (searched  in  DIALOG  Ftle64) 

QOO*  Effects  of  controlled  lung  hyperinflations,  before  and 

after  endotracheal  suctioning^  on  the  cerebrovascular 
status  of  adults  with  severe  closed  head  injuries 
(searched  in  DIALOG  Filel54) 

Q005  Rules-of-thumb,  industry  by  industry    ( searched  in  DIALOG 

Fllel48) 

Prevention  of  carbon  dioxide  crystal  growth  on  the  interior 
surfaces  of  reactors     (searched  in  DIALOG  FlleS) 

Factors  which  impede  strategic  human  resource  management 
(searched  in  DIALOG  File75) 

QP08  Effects  of  an  aerobic  interval  training  program  oii  the 

physical  and  psycho-social  health  of  menopausal  women 
(searched  in  DIALOG  File  154) 

Alternatives  for  delivery  of  human  services  other  than  the 
classical  model  of  individual  casework  in  an  agency  based 
office    (searched  in  DIALOG  File37) 

QOiO  Motivations  of  adults  choosing  to  discontinue  chemotherapy 

(searched  in  DIALOG  Filel54) 

Q^ll  Psychd-emotipnal  arid  psycho^social  responses  of  parents  and 

surviving  siblings  to  an_lnfant*s  death  due  to  Sadden 
Infant  Death  Syndrome  (SIDS5     (searched  in  DIALOG  Fiiel54) 

Q012  Chemical  reactivity  of  silicon  carbide  and  silicon  nitride 


ceramic  powders  at  low  (room)  temperatures  especially  In 
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aqueous  environments    (searched  in  DIALOG  Filel35 


Q013  Definition  aiid  measurement  of  effectiveness  in  noti-profit 

human  service  organizations    (searched  in  BiidjQG  ?iiei5) 

QQIA  Changes  in  the  function  of  hospital  information  systems  due 

to  the  advent  of  prospective  pajrment  systems    (searched  in 
DIALOG  FilelSl) 

Q015  Occurrences^  causes,  treatment »  and  prevention  of 

r<5troientai  fibropiasia    (searched  in  DIALOG  Filel54) 

Q016  Retirement  activities  including  pre-retirement  indicators 

of  retirement  activity  patterns    (searched  in  DIALOG 
File! 1 5 

Q017  Ptmps  and  control  systems  for  drug  delivery  in  animal  

experiments  and  clinical  applications     (searched  in  DIALOG 
Files) 

Q618  Managerial  competencies  especially  as  applied  to 

physician-managers    (searched  in  DIALOG  FilelS) 

Perceived  impact  of  the  1977  Institute  of  Internal  Auditors 
Standards     (searched  in  DIALOG  File75) 

Q02b  Presentation  of  financial  statemehtSi  especially  the 

disclosure  requirement  fora  of  the  SEC    (searched  in  DIALOG 
;  FilelS) 

Q021  Social  support  networks  and  the  physical  and  mental  health 

of  never  married  older  women    (searched  in  DIALOG  Fiie37) 

Q022  Space  commercialization  forecast     (searched  in  DIAOG 

FilelOS) 

Q023  Sintered  powder  metal  or  powder  metal  parts  infiltrated 

with  copper  or  bronze    (searched  in  DIALOG  File32) 

Q024  Meaning  of  the  cat  in  Italian  Renaissance  (1450-1600) 

religious  paintings     (searched  in  DIALOG  Filel91) 

Q02S  Relationship  between  oral  and  written  language  of  basic 

writers     (searched  in  DIALOG  Filel) 

Q026  Policies  of  creating  administrative  agencies  for  purposes 

of  compensating  industrial  workers  accidentally  killed  or 
injured  in  Ohio  or  Ontario  from  191S  to  1935    (searched  in 
DIALOG  File38) 

Q027  Principles  and  design  of  miniature  high  pressure  sensors 

(searched  in  DIALOG  Filel3) 

Q028  History  from  1800  of  University  Circle  in  Cleveland 
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focusing  on  philanthropy,  city  planningj^  and  public  vs. 
private?  development    (searched  in  DIALOG  File38) 


Q029  Firing  or  sintering  of  ceramic  material  using  microwave 

radiation    (searched  iii  DIALOG  FileS) 

Creativeeyasion  of  censorship  in  South  Africa  (searched 
in  DIALOG  File71) 

Q03i  Budgeting,  especially  automated  acquisition  budgeting,  in 

law  libraries    (searched  in  DIALOG  File61) 

Q032  Engineering  properties  and  various  utilizations  of  fly  ash 

as  a  construction  material    (searched  in  DiM.OG  FileS) 

Q033  Volume-averaged  equations  used  to  determine 

friction-factors  of  2-phase  slurry  flow  in  pipelines 
(searched  in  DlMiOG  FileS) 

Q034  Expert  systems  directed  by  the  user  and  not  by  an  inference 

engine    (searched  in  DIALOG  Filel3) 

Q035  ^sic  therapy  for  the  chronically  ill,  especially  cancer 

patients    (searched  in  DIALOG  Filel54) 

Q036  industrial  policy  in  Austria  and  Western  Europe  related  to 

technological  innovation,  restrticturirig  of  induitry^  the 
EEC,  and  corporatism    (searched  in  DIALOG  File90) 

Q037  Training  of  employees  oii  the  right  to  know  (RTK)  laws,  OSHA 

hazard  compliance  laws,  chemical  safety ^  and  handling  of 
hazardous  materials    (searched  in  DIALOG  iilel6) 

Q03S  Future  of  document  acquisition^  cataloging^  storage,  and 

information  dissemination  in  the  automated  technical 
reference  libr^try    (searched  in  DIALOG  Flle6l5 

Q039  Environment  of  a  corporation  as  it  affects  organizational 

structure    (searched  in  DIALOG  FilelS) 

Q0*0  Known  or  proposed  techniques  for  bacterial  cloning  and  the 

commercial  activity  surrounding  the  technology  (searched 
in  DIALOG  Filel6) 
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Tabte  8^ 


TYPE  OF  SEARCH  AND  RESTRICTIONS  AS  REQUESTED  BY  THE  USERS  FOR  TSEOk 
QUESTIONS  (N  question  -40) 


Number  of 
Questions 


Type  of  Search  Requested 
Precise  Search 
Broad  Search 
Not  Specified 


2 
36 
2 


Restrictions  for  Language  of  Articles  Retrieved 

English  Only  25 
Any  Language  15 


Years,  to  be  Searched 
Last  5  Years 
No  Limits 

Other  Limits  Specified 

Within  Last  3  Years 
Within  Last  10  Years 
Within  Last  25  Years 
Within  Last  40  Years 


4 
25 

1 

4 
5 
1 


DIALOG  Databases  Suggested 
None 

1-2  Files  Speficied 
3-4  Files  Specified 


32 
5 
3 
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Table  8-5 


SUMHARY  OF  VAtHES  IN  COpEXf  pASHRES  ASSIGNED  BY  USERS  IN  RELATION  TO 
THEIR  QUESTIONS  (N  questions  -40) 


Number  of  Users  Assigning  the  Given  Value 


Context 
Measure 

1 

2 

3 

4 

5 

Problem  Definition 

1 

6 

10 

13 

10 

Intent 

6 

12 

7 

9 

Public  Knowledge 

1 

9 

6 

12 

Internal  Knowledge 

0 

4 

18 

It 

3 

■  

Table  8-6 


SUMMARY  OF  VALUES  IN  CONTEXT  MEASURES  ASSIGNED  BY  PROJECT  SEARCHERS  FOR 
TIffi  QUESTION  TlffiY  SEARCHED.    This  represents  the  project  searcher's  view 
of  the  context  as  opposed  to  the  user's  view  of  th^sir  own  question 
context  as  summarized  in  Table  (N  questions  »  40) 


Number  of  Questions  Assigning  the  Given  Value 


By  Project  Searchers 

Context  " 

Measure  12  3  4  5 

Problem  Definition  6             7  7  11  9 

intent  9           io  7  7  7 

Public  knowledge  10           10  8  7  5 

Internal  Knowledge  24           10  4  0  2 
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Table  8-7 


SUFMftRY  OF  VAtUES  ON  TWO  CONTEXT  MEASURES  ASSIGNED  BY  OUTSIDE  SEARCHERS 
IN  RELATION  TO  THEIR  SEARCHES  (N  searches  »  200  -  (40  questions  X  5 
searches  for  each)) 


Number  of  Searches  Assigned  the  Given  Value 
By  Outside  Searchers 


Context  Not 

Measure  Assigned  12  3  4  5 

Public  Knowledge  24  6           10  58  56  46 

Internal  Knowledge  12  71           52  34  28  3 
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9.  BAdCGROtJND  DATA  ON  SEARCHERS 


?e-Scores 


9.i.i  bvarv4w 


The  study  involved  the  fdllbwing: 

36      outside  iearchers  (each  of  whom  searched  5  or  6  questions 
of  the  40  questions  submitted) 

3      project  searchers  (each  of  whom  conducted  4  project 
searches) 

1      analyst  used  on  question  classification  only 


40  searchers 

[As  explained J  the  project  searches  were  done  to  test  search 
strategies  devised  from  different  sources;  they  were  labeled  project 
search  1^  2^  3 j>  and  4;  these  were  based  on:  1.  taped  problem 
statetment  only;  2*  taped  problem  statement  plua  the  written 
5^??^$9?5  from  the  written  question  statement  pttly  (no 

elaboration);  4.  terms  from  the  written  question  plus^  elaboration 
through  an  appropriate  thesaurus.] 

Table  9^J^  is  used  to  list  outside  and  project  searchers  by  their 
code  (instead  of  by  name)  so  that  the  scores  listed  in  Table  9-2,  if 
desired,  could  be  associated  with  the  type  of  searchers. 

Table^-^2  provides  the  test  scores  for_ 40  searchers.  Scores 
were  given  for  the  Remote  Associates  Test  (RAT)^  EmplbyeeAptitude 
Survey  XEAS),  and  seven  scores  wlthih  the  Learning  Style  Inventory 
Test.    These  tests,  together  with  the  meaning  of  score  numbers  is 
given  next. 


9.1.2  Remote  Associates  Test 

The  Remote  Associates  Test  is  a  test  of  semantic  association. 
The  test  presents  subjects  with  sets  of  thr^  and  asks 

the  subject  to  find  a  fourth  word  that  is  related  to  (or  has 
something  in  co^bn  with)  all  three  stimulus  words,  as  in  the 
following  examples: 

cookies         sixteen         heart    [sweet] 

poke  go  moiasses    [slow] 
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surprise 


line 


birthday 


[party] 


skunk 


kings 


boiled 


[cabbage] 


The  test  consists  of  30  such  matches  to  be  made  in  20  minutes* 
The  score  is  a  straight  count  of  right  answers;  thus,  RAT  score  o£  16 
means  that  the  subject  tested  had  15  correct  match  words  out  of  a 
possible  30. 

Table-  9-3  provides  the  distribution  of  RAT  scores  and  mean 
values  for  40  searchers.    It  shows  that  there  are  two  searchers  with 
5  right  matches,  4  with  6  right  matches,  etci    The  mean  number  of 
right  matches  for  40  searchers  was  13 i 

Note  thatthe  distribution  in  Table  9-3  shows  two  distinct  peaks 
at  10  and  at  18.    Thus,  the  mean  is  not  typical  and  the  standard 
deviation  does  not  represent  the  width  of  a  "central  peak".  (The 
searcher  code  numbers  can  be  viewed  as  a  histogram.)    This  is  true  of 
many  of  the  distributions  found  in  this  study.    They  were  not  nornal 
distributions,  i.ei,  bell  shaped,  and  the  means  have  to  be 
interpreted  with  caution. 


9.1.3  Employee  Aptitude  Suirvey 


The  Employee  Aptitude  Test  (EAS)  is  a  series  of  lOshort  tests 
that  are  used  In  business  and  industry  for  personnel  selection.  We 
have  selected  only  one  of  these  10  tests:  the  Sjnnbblic  Reasoning 
Test.    This  is  a  30  item  test  done  in  5  minutes i  designed  to  test 
deductive  inference.    It  uses  inequalities  as  test  items.    Each  of 
the  30  items  specifies  a  relationship  of  "A"  to  "B"  to  "C"  as  in  the 
following  example: 

A'^  B<  C 

and  asks  if  therefore 

A?^  C  is  true  I  false,  or  dont '  know. 

The  test  is  also  scored  as  a  straight  count  or.  correct  answers. 
Thus,  a  score  of  10  means  that  the  subject  tested  identifted  10 
; correct  answers  out  of  30. 

Table  9-4  provides  the  distribution  for  EAS  scores  and  mean 
values  for  40  searchers  i    The  mean  n^ber  of  correct  answers  for  40 
searchers  was  10,  but  the  distribution  also  has  more  than  one  peak. 

9.1.4  Learning  Style  Inventory  \ 

  Theory.    The  Learning  Style  Inventory  (LSI)  is  based  on  a  theory 

that  learning  is  an  integrated,  four  stage  process  that  Involves  the 
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use  of  four  different  cognitive  modes  [5-3]: 


!•  Concrete  Experience  (CE)^  thie  ability  to  involve  oneself 
fully,  openly  and  without  reservation  in  hew  experiences; 

2.  Reflective  Observatlbn  (RO),  the  ability  to  reflect  on  and 
observe  these  experiences  from  many  different  points  of  view; 

3.  Abstract  Conceptual tzat ion  (AC),  the  ability  to  generate 
concepts  that  integrate  observations  into  logically  sound  theories; 

^»  Active  Experimehtattoh  (AE),  the  ability  to  apply  those 
theories  to  solve  problems  and  make  decisions* 


The  model  is  usually  represented  as: 


CONCRETE 
EXPERIENCE 
(CE) 


TESTING  THE  OBSERVATION 
IMPLICATIONS  OF     (AE)  (RO)  AND 

ONE'S  THEORIES  REFLECTION 


FORMATION  OF 
ABSTRACT  CONCEPTS 
AND  GENERALIZATIONS 

_  The  Learning  Style  Inventory  (LSI)  measures  individual 
preferences  for  each  of  the  four  basic  learning  modes  (CE,  RO,  AC, 
AE)  and  places  an  individual  on  a  grid  formed  with  Concrete 
Experience  (CE)  and  Abstract  Cbnceptuaiization  (AC)  forming  the 
Y  axis  and  Reflective  Observation  (RO)  and  Active  Experimentation 
(AE)  the  X  axis,  as  if  they  were  drawn  through  the  above 
representation  of  the  learning  model  up  and  down  and  from  left  to 
right . 


The  theory  recognizes  that  learning  requires  the  use  of  all  four 
modes,  t>ut  that  most  people  develop  a  preference  for  each  contrasting 
pair,  a  preference  that  is  known  as  a  learning  gtyle>    There  are  four 
predominant  styles  of  learning,  shown  schematically  in  the  following 
diagram: 
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ACTIVE 

EkPERIMENTATION 


CONCRETE 

ACCOMODATION 

DIVERGENCE 

EXPERIENCE 

ABSTRACT 

CONVERGENCE 

ASSIMILATION 

CONCEPTUALIZATION 

A  preference  for  the  modes  of  concrete  experience  and  active 
experimentation  Indicates  ah  accomodative  style  of  learning.  The 
theory  explains  that  accomodators  tend  to  solve  problems  In  an 
intuitive  trlal-and-error  way^  relying  heavily  on  others  for 
infonnatibn  rather  than  on  their  own  analytical  ability. 

A  Preference  for  the  modes  of  concrete  experience  and  reflective 
observation  indicates  divergent  style  of  leartiing.    bivergeri3  excel 
at  the  Ability  to  view  concrete  situations  from  many  perspectives  and 
to  organize  diverse  elements  into  a  single  meaningful  gestalt. 

 A  preference  for  abstract  conceptualization  and  active 

experimentation  defines  the  convergent^  style  of  learning. 

A  converger's  knowledge  is  organized  in  such  a  way  that  he  can  focus 

it  on  specific  problems  through  hypothetical  deductive  reasoning. 

 Finally,  preferences  for  abstract  cpriceptualizatibn  and 

reflective  observation  define  the  assimilative  learning  style. 
Assimilators  excell  at  inductive  reaspriihg,  assimilating  disparate 
observations  into  integrated  ejcplahations . 

i^Btr.    Itearnlng  Style  Inventory  (LSI)  is  a  test  based  on  the 
above  theory  of  learning.    It  is  used  to  measure  individual 
preference  for  each  of  the  four  basic  leaxming  modes  (concrete 
experience  (CE),  reflective  observation  (RO),  abstract 
cbhceptualizatioh  (AC),  and  active  experimentation  (AE)).    The  test 
form  asks  the  respondent  to  rank  in  order  from  1  to  4  a  series  of 
fbiir  statements  in  response  to  a  question  on  how  he/she  learns.    A  20 
minute  time  period  is  allowed.    There  are  no  right  or  wrong  answers. 
There  are  12  sets  of  four  statements  to  rank.     Two  examples  are: 

When  I  learn: 

i  like  to    -  1  like  to 

think  about  be  doing 

ideas  things 


Rational    A  chance  tb 

theories  try  out  and 

practice 


  I  like  to    I  like  to 

deal  with  watch  and 

my  feelings  listen 

i  learn  best  from: 

 Personal  Obser- 

-        relations  vation 
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The  LSI  yields  a  total  of  six  scores:  one  raw  score  for  each  of 
the  four  basic  modes  and  two  composite  scores.    The  raw  scores  are 
obtained  by  summing  the  ranks  indicated  down  the  columns  of  the  LSI. 
Each  column  represents  a  separate  learning  ability,  arranged  from 
left  to  right  across  the  page  in  the  following  order:  concrete 
experience  (column  l)^  reflective  observation  (column  2),  abstract 
conceptualization  (column  3)  and  active  experimentation  (column  4)* 
The  raw  scoring  is  done  by  simply  adding  all  the  rank  numbers  given 
by  a  respondent  in  a  column i    Because  each  value  for  a  rank  can  be 
between  1  and  4,  and  there  are  12  items  in  a  column,  the  maximum 
score  is  48  and  the  minimum  is  12. 

 The  two  combination  scores  are  obtained  by  subtracting  a 

respondent's  concrete  experience  score  from  the  abstract 
conceptualization  score  (AC-CE),  and  the  reflective  observation  score 
from  the  active  experimentation  score  (AE-RO).    The  combination 

determine  a  respondent Vs  learning  style.  They 
indicate  the  extent  to  which  the  individual  emphasizes  abstractness 
over  concreteness  (AC-CE)  and  the  extent  to  which  he  or  she 
emphasizes  action  over  ref lection  (AE-RO).    The  combination  score 
values  can  range  from  -36  to  +36. 


In  addition y  a  graph  is  made  for  the  combination  scores  for 
visual  identificationpf  the  person's  learning  style.    The  X  axis  is 
used  to  place  the  AC-CE  composite  scores,  and  the  Y  axis  for  the 
AE-RO  scores.    The  point  of  interception  between  these  two  scores  is 
used  to  place  a  respondent  in  one  of  the  four  quadrants,  identifying 
the  learning,  styles.    The  quadrants  representing  the  four  types  of 
learning  styles  (labeled  1  to  4  in  Tables  9-2  and  9-6^)  are  as  follows 
(see  explanation  for  each  type  in  the  .preceeding  section): 

1.  Converger:  left  lower  quadrant 

2.  Diverger :  right  upper  quadrant 

3.  Assimilator:  right  lower  quadrant 

4i  Accomodator :  left  upper  quadrant 

 A  person  with  a  score  of  zero  on  either  AC-CE  or  AE-RO  is 

considered  as  indeterminate*    The  closer  a  data  point  is  to  the  point 

^^9^^         more  balanced  the  respondent's  learning  style, 
^^?_f??^^®^_?^?o  j^^y J^o^^s^t  the  more  emphasized  are  the 
characteristics  of  the  given  learning  style  within  the  quadrant. 

 As  mentioned,  Table_9^  provides  the  raw  scores  for  the  Learning 

Style  Inventory  for  all  searchers^  together  with  composite  scores  and 
the  type  of  learning  style  derived  from  the  graph  in  Figure  9-1. 

table  9*^5  provides  the  range  and  mean  of  the  four  basic  scores 
(CE,  RO,  AC,  and  AE)  and  the  combined  scores  (AC-CE  and  AE-RO)  for 
all  the  searchers.    As  can  be  seen  the  abstract  conceptualization 
mode  of  learning  has  the  highest  mean  and  the  concrete  experience  the 
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^Iguge-a-^I  is  the  graphic  representation  for  the  coiubine^  scores 
of  all  searchers.     (The  graph  is  derived  from  the  combined  scores  in 
Table-g-^a. )    It  slAows  the  place  of  individual  searchers  in  different 
quadrants i  each  quadrant  representing  a  different  style  of  learning. 

_      Table  9-6  sijnmari^es  the  data  from  Etgure  9-1  showing  the  number 
of  searchers  with  any  given  learning  style  and  lists  their  search 
codes.    As  can  be  seen,  42%  of  the  searchers  were  convergers,  25% 
were  assimilatbrs,  23%  were  accomodators,  5%  were  divergers,  and  5% 
were  indeterminate. 


9.2  Online  Background^^f- Searchers 


 The  searchers  were  asked  (Form  3  Appendix  H)  to  answer  several 

questions  about  their  online  experience.    The  next  two  tables  are 
compiled  from  this  data. 

 Table  9-7  shows  the  DIALOG  databases  most  frequently  used  by 

searchers.    (The  list  of  selected  databases  is  given  in  Form  3, 
Appendix  H . )    Each  searcher  was  asked  to  indicate  seven  databases 
from  most  frequently  used  (rank  1)  to  least  used  (rank  7).    From  the 
table  we  can  8ee_that_the  most  frequently  used  databases  are:  ERIG 
(education),  COMPENDEX  J engineering),  Psychlnfo  (psychology),  PfS 
PROMPT  (business),  ABI/INFORM  (business),  MEDtiNE  (medicine),  NTIS 
(government  reports) i  PTS  F&S  Indexes  pusiness).  Sociological 
Abstracts,  INSPEC  (science  and  engineering)  and  Magazine  index 
(popular  periodicals).    A  number  of  other  databases  from  a  variety  of 
fields  were  mentioned.    Thus,  the  searchers  as  a  group  had  a  diverse 
database  background. 

Table  9-8  shows  the  frequency  of  searching  of  DIALOG  as 
Indicated  by  the  40  searchers i    As  can  be  seen  30%  reported  using 
DIALOG  daily,  and  another  33%  twice  a  week.    Seven  percent  reported 
using  DIALOG  once  a  week^  while  about  22%  use  it  less  than  twice  a 
Ttiotitb.    Thus,  some  70%  of  the  searchers  use  DIALOG  at  least  once  a 
week.    In  other  words,  these  were  experienced,  professional 
searchers • 
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CODE  NUMBERS  USED  FOR  VAFrC\':H  tJARCHERS  AND  FOR  PROJECT  SEARCHES  (for  use 
In  conjunct  I/:^; a  Srtth  TabJo.  /-/ ,) 


Type  of  Searcheie 

Number 

Code  Designation 

Outside  Searchers 

36 

SOOl  to  S017;  S021;  S023  to 
S041 

Project  Searchers 

3 

S018,  S019,  S020 

Analyst  for  Question 
Classification 

1 

Designated  Project 
Number  Devised 

Search 
From 

Code  Designation  for 
Project  Search  Plus  Project 
Searcher 

-               -  - 

1              User's  taped  problem 
statement  only 

S118,  3119,  S120 

2  User's  taped  problem  S2i8,  -'^^i'^  S220 

statement  plus  written 
question 


3  Words  from  written  S318,  S319,  3320 

question  only  ^no 
elaboration) 


4  Terms  from  written  3418,  3419,  3420 

question  plua  thesaurus 
elaboration 
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Table  9-2 


COGNITIVE  TEST  SCORES  FOR  ALL  SEARCHERS 


Record # 

i 
2 
3 
4 
5 
6 
7 
S 
9 
10 
ii 
12 
13 
14 
15 
16 
17 
IS 
19 
20 
21 
22 
23 
24 
25 
26 
27 
28 
29 
30 
31 
r> 


33 
34 
35 
36 
37' 
38 
39 
4CJ 


SEAR 

ddi 

002 

003 

004 

005 

006 

007 

008 

009 

010 

Oil 

012 
013 

014 

015 

016 

017 

018 

019 

020 

021 

022 

023 


024 

025 

026 

027 

028 

029 

030 

032 

033 

034 

035 

036 

037 

038 

039 

040 

041 


RAT 

EAS 

CE 

RO 

AG 

AE 

AC  rCE 

AE-RO 

LSI 

16 

11 

27 

30 

23 

40 

-4 

10 

4 

14 

la 

13 

31 

48 

28 

35 

—3 

3 

IS 

7 

33 

17 

31 

39 

-2 

22 

4 

IS 

21 

19 

35 

31 

35 

12 

0 

0 
3 

13 

IS 

15 

32 

42 

31 

27 

-1 

19 

21 

22 

33 

41 

24 

19 

-9 

■  3 

11 

_  5 

36 

32 

16 

36 

-20 

4 

4 

10 

14 

23 

32 

30 

35 

7 

3 

1 

1 Y 

5 

21 

30 

29 

40 

8 

Id 

1 

11 

13 

18 

35 

35 

32 

17 

—3 

3 

lo 

17 

14 

30 

48 

28 

34 

-2 

3 

lo 

10 

19 

38 

41 

22 

22 

-16 

3 

10 

12 

17 

34 

39 

3d 

22 

-4 

3 

19 

14 

36 

44 

25 

15 

-11 

-29 

2 

8 

5 

17 

31 

48 

24 

31 

-7 

3 

9 

3 

28 

23 

32 

37 

4 

14 

1 

lu 

5 

42 

19 

30 

29 

-12 

10 

4 

-M- 

6 

9 

21 

32 

44 

23 

23 

-9 

3 

18 

24 

12 

48 

36 

24 

24 

1 

9 

11 

17 

19 

48 

36 

31 

17 

1 

8 

8 

29 

27 

3b 

34 

1 

_7 

1 

28 

12 

23 

Is 

34 

45 

11 

22 

1 

6 

4 

17 

29 

45 

32 

28 

3 

1 

18 

9 

26 

21 

34 

39 

S 

IS 

1 

13 

11 

37 

14 

24 

45 

-13 

31 

4 

10 

8 

25 

33 

19 

43 

-6 

10 

4 

5 

5 

25 

25 

24 

36 

-1 

11 

4 

14 

3 

33 

31 

25 

31 

-8 

0 

0 

P 

9 

28 

19 

46 

27 

18 

8 

1 

16 

5 

31 

25 

24 

4d 

-7 

15 

4 

21 

5 

16 

21 

44 

39 

18 

18 

1 

11 

•  5 

24 

24 

26 

46 

2 

22 

1 

18 

r 

34 

24 

17 

45 

-17 

21 

4 

11 

l5 

31 

19 

33 

37 

2 

18 

1 

B 

l8 

37 

41 

25 

17 

-12 

-21 

2 

l7 

14 

22 

33 

30 

35 

8 

2 

1 

18 

16 

18 

37 

27 

38 

9 

1 

•1 

12 

14 

22 

35 

42 

21 

2d 

-14 

3 

10 

9 

27 

23 

28 

42 

1 

19 

1 

19 

8 

21 

28 

36 

35 

15 

7 

■1 

LEGEND: 

SEAR_NO=S earcher 

NutnbiBr 
RAT  «  Remote 

Associates  Tea 
EAS  «  Employee 

Aptitude  Surve 
CE  «  Concrete 

Experience 
RO  «  Reflective 

Observation 
AC  «  Abstract 

'Conceptualizati 
AE  «  Active . 

Experimentation 
AC-CE  »  Difference 
between 
AC  and  CE 
AE-RO  «  Difference 
between 
AE  and  RO 
LSI  «  Learning  Style 
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tahie-  9-3 


DISTRIBUTION  6)F  REMOTE  ASSOCIATES  TEST  (RAT)  SCORES  FOR  M.E.  SEARCHERS 
(N  searchers  *  40;  possible  score  range  from  1  to  30) 


Number  of 

Score  Searchers      Searcher  Code  Nimber 


5  .  2  S027,  S036 

6  4  S018,  S019,  S023,  3029 
8  2  S0i5,  S021 

.9  2  S016,  S020 

10  5  S008,  Sdl3,  Sdl7,  S026,  sb40 

11  4  S007,  sold,  Sd33,  SOBS 

12  1  Sd39 

13  2  Sdd5,  S025 

14  2  sdd2,  S028 

15  1  S0d4 

16  3  sdOl,  S0b3,  S030 

17  2  S009,  S037 

18  5  soil,  S012j  S024,  S034,  S038 

19  3  S006,  S014,  S041 
21  i  S032 

28  i  S022 


MEAN  :  3,3.63 
STANDARD  DEVIATION  :  5.2 
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Table  9-4 


DiSTRiBUTiON  OF  EMPLOYEE  APTITUDE  SURVEYJEAS)  SCORES  FOR  ALL  SEARCHERS 
(N  searchers  -  40 j  possible  score  range  from  1  to  30) 


Number  of 

Score  Searchers^    Searcher  Code  Number 


3  1  S006 

4  i  S023 

5  8  S007i  SD09,  S015,  S017,  S027,  3030,  3032,  3033 

7  2  8003,  S034 

8  4  S021,  S026,  S028,  S04i 
.9  4  3018,  S024,  S029,  3040 

10  1  S012 

11  3  SOOl,  S020,  3025 

12  2  S013,  3022 

13  1  SOlO 

14  4  3008,  S0i4,  Sd37,  3039 

15  2  SO05,  3035 

16  1  3038 

17  1  3011 

18  •  3  3002,  Sbl9^  3036 
21  2  3004,  Sbb6 


M2AN  :  10.60 
STANDARD  DEVIATION  :  4.965 
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Tafele  9-5 


MEAN  SCORES  FOR  LEARNING  STYLE  INVENTORY  (LSI)  FOR  ALL  SEARCHERS 

(N  searchers  >  40;  possible  range  for  CE,  lElO,  AC,  and  AE  from  12  to  48; 

for  AC-CE  and  AE-RO  from  -36  to  +36) 


Standard 

Learning  Mode  Minimum  Maximum  Mean  Deviation 

(CE)  CONCRETE  EJEPERIENCE  13  42  24.7  7.3317 

(R0)  REFLECTiVE  OBSERVATION  12  44  27.9  7.4482 

(AC)  ABSTRACT  CONCEPTUALIZATION  16  48  33.55  9.4134 

(AE)  ACTIVE  EXPERIMENTATION  15  46  33.675  7.7736 

Combined  Scores 

AC-CE  -20  35  12.850  11.6102 

AE-RO  -29  31  9.725  10.6168 

LS  -  Learning  Style  (Quadrant)        1  4  2.175  1,3375 
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table  9-6 


biSTRIBUTICJ  r>.?  LEARNING  STYLES  FOR  ALL  SEAS_^gRS,    Derived  from  combined 
scores  as  plotted  in  Figure  9-1,     (N  searchers  -  40;  N  learning  ecyles  » 
4  plus  Indeterminate) 


Learning  Style  Number  of 

and  Code  Used  Searchers        Searcher  Oode  Number 


0 

-  INDETERMINATE 

2 

S004, 

S028 

1 

-  CONVERGER 

17 

S008, 

S009, 

S016, 

S019, 

S020i 

S021 

S022, 

S023, 

S024j 

5029* 

S032, 

S033, 

S035, 

S037, 

S038, 

S040, 

S041 

2 

-  DIVERGER 

2 

S014, 

S036 

•i 

-  ASSmitATOR 

id 

S002, 

SOOSi 

5006* 

SOlO, 

soil, 

S012, 

8013* 

S015* 

S018, 

SOS  9 

4 

-  AeeOMODATOR  , 

9 

SOOlj 

S003i 

S007, 

S017, 

S025, 

S026, 

S027, 

3030, 

S034 
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Figure  9--1 


®*^H±^  REPRESENTATION  FOR  THE  COMBINED  SCORES  ON  THE  LEARNING  STYLE  TNVFNTOWY 
PLACING  EACH  SEARCHER  IN  A  QUADRANT  INDICATING  A  GIVEN  LEArS/s^I  (H^rcLrs-AO 


'J  M 


HI 


< 


95 
O 

•-^  : 

<  I 

ii 

i 


A  -  ACCOMMbbATOR 


0 


o1 


-30 


2  -  DIVERGER 


.---20 


X  d3V 


10 


+30     o  31  «+2ox  o  3  5-     +1;:'  : 


•10 


-20 


_-30 


+10 


1  -  CONVERGER 


,,-1-20 

XO/3 


X006, 


-1-3(7  xo/r 
Xori  3  -  ASSIMILATOR 


ACTIVE 

EXPERIMENTATION  (iffi) 


AE.-RO 


REFLECtiVE 

OBSERVATION 
(RO) 
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Table  9-7 


DIAtOe  SATABApS  AS  MOST  OFTEN  SEARCHED  BY  SEARCHERS 

(N  searchers  "4b;  they  indicated  up  to  7  databases;  most  frequently 

searched  rank  »  1,  least  frequently  rank  »  7) 


Number  of  Searchers 
Selecting  Frequency  Rank 


File 

NumbeF  and  Name 

■    

1 

X 

0 

c. 

3 

c 
J 



7 

 , 

UU  J. 

ERIC 

1  n 

D 

i. 

A 
U 

z 

COMPENDEX 

Q 

J 

a 

i 
I 

0 

u 

u 

nil 

Uii 

PflVnhTtrFn 

*T 

■J 

J 

0 
J 

u 

U 

1 

1 

016 

PTS  PROMPT 

A 
•+ 

/ 

*♦ 

i 
X 

U 

U 

015 

- 

AB I /INFORM 

i 
1 

i. 

1 

1 
1 

154 

MEDLINE 

u 

r\ 
J 

1 

U 

006 

NTIS 

2 

5 

3 

3 

1 

0 

4 

018 

PTS  F&S  INDE3BES 

2 

2 

2 

1 

b 

2 

0 

132 

STANDARD  AND  P0dR*S  NEWS 

I 

0 

0 

0 

1 

2 

0 

037 

SOCIOLOGiCAL  ABSTRACTS 

I 

1 

2 

0 

2 

2 

2 

032 

1 

1 

1 

1 

1 

0 

1 

013 

iNSPEC 

0 

4 

2 

3 

2 

3 

1 

218 

NURSING  AND  AttlED  HEALTH 

0 

2 

0 

0 

0 

0 

0 

047 

MAGAZiNjS  INDEX 

0 

1 

2 

3 

4 

2 

3 

148 

TRADE  ^  INDUSTRY  INDEX 

0 

1 

0 

4 

0 

1 

0 

086 

^ffiNTAL  HEALTH  ABSTRACTL 

0 

1 

0 

I 

0 

1 

0 

075 

MANAGEMENI  CONTENTS 

0 

1 

0 

2 

1 

0 

033 

WORLD  ALUMINUM  ABSTRACTS 

0 

1 

0 

0 

0 

0 

0 

005 

BIOSIS 

0 

0 

1 

1 

0 

0 

1 

111 

NATIONAL  NEWSPAPER  INDEX 

0 

0 

3 

2 

2 

3 

2 

161 

OCCUPATIONAL  SAFETY  AND  HEALTH 

0 

0 

1 

1 

0 

0 

0 

071 

MLA  BIBLIOGRAPHY 

0 

0 

1 

1 

0 

0 

0 

049 

PAIS  INTERNATIONAL 

0 

0 

1 

0 

1 

0 

0 

014 

ISMEC 

0 

0 

1 

0 

1 

0 

0 

Additional  DIALOG  databases  ranked  4   .6  7  by  1  or  2  searchers  were: 
(4)  036,  035;  (5)  Oi      051,  089*  171,  076,  233,  007,  064;  (6)  291,  275, 
039,  040,  233,  064;  {/     .39,  097,  238,  038 


I 
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Table  9-8 


FREQUENr'  of  DIALOG  SEARCHING  BY  SEARCHERS  IN  THE  STUDY  (N  searchers  »  40 
composed  of  36  outside  searchers,  3  project  searchers  &  1  analyst  for 
question  classification) 


Frequency  of 


Searchers  at 
That  Frequency 


Searcher 


Use  and  Code 

No. 

z 

Number 

1  ■  Less  than  twice 
a  month 

9 

22.5 

S012, 
S023, 

8013, 
S024, 

S016. 
3028, 

5018, 
3032 

3019, 

2,  Twice  a  month 

3 

7.5 

S015, 

S022, 

3034 

3.  Once  a  week 

3 

7.5 

soot, 

S020, 

3038 

4.  Twice  a  week 

13 

32.5 

S002, 

soil, 

S03d, 

S006, 
Sdl7, 
3036, 

3007, 
S025, 
3037 

3008, 
S027, 

3010, 
3029, 

5.  Daily 

i2 

30 

S003, 
S021, 
S039, 

S004, 
3026, 
3040 

3005, 
3031, 

S009, 
3033, 

3014, 
3035, 
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10.  EFFiCiENCY  CHARACTERISTICS  OF  SEARCHES 


A  series  of  miaasures  (defined  in  Section  5. 5.5)  were  used  to 
characterize  a  search  on  the  basis  of  efficiency:  time  (online  and 
offline)  and  structure  (number  of  commands ^  search  terms,  and 
cycles).    Both  efficiency  and  structure  have  bearing  on  and  can  be 
translated  into  costs. 

The  elements  of  time  and  structure  characterize  a  search 
Independently  from  evaluations  of  retrieved  Items,    these  can  be 
derived  and  studied  without  viewing  the  results »    fhus^  they  are 
ipresented  here  before  the  results  on  items  retrieved  and  the 
effectiveness  measures.    The  efficiency  measures  are  objective 
measures  of  a  search  and  are  used  for  correlation  with  other 
measures,  particularly  those  involving  effectiveness. 

In  the  u  iree  tables  in  this  chapter  and  all  of  the  tables  which 
follow,  we  are  using  the  abbreviation  for  variables        defined  in 
Chaptar  7,  Table  7-i ,  where  a  detailed  definition  of  each  variable 
and  measure  used  is  presented,    in  other  words,  instead  of  a  repeat 
of  the  definition  of  the  variables  and  measures*  a  reference  i3  made 
to  the  code  book  and  only  the  title  of  th-?:  n^aasure  is  given. 

 ^^^^^s  in  this  chapter  contain  the  mieani^  standard 

deviation,  and  the  minimum  and  maximum  values  for  the  five  efficiency 
measures  for  the  searches. 

Table  contains  efficiiBncy  measures  for  the  _20C  searches 

done  ijy  the  outside  searchers  (40  questions  X  5  searches  of  each). 

TahXQ^li)^2r  contains  efficiency  measures  for  the  160  searches 
done  33  project  searches  (40  questions  X  4  project  searches  each). 

TafeliV  10-3  contains  efficiency,  measures  for  all  360  searcl.Gs, 
combining  outside  and  project  searches. 

As  can  be  seeh^  the  range  is  quite  wide  and  the  standard 
deviation  quite  large.    Differences  exist  between  the  outside  and 
project  searches^  however,  the  differences  are  not  large  and  they  are 
not  statistically  significant. 

As  can  be  seen,  on  the  average,  per  search,  the  searchers  used 
in  the  neighborhood  of: 

...  13  commands 

...  3  cycles 

...  10  search  terms 

...  1^  minutes  of  online  time 

• • •  l3  minutes  of  preparation  time 
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•  •  •  2^  tnlhutes  total  time 

But,  the  ranges  for  a  search  vai*ied  from: 

•  2  to  50  cbimnands 

•  •  •  1  to  14  cycles 

i  to  61  search  terms 

• • •  half  a  minute  to  74  minutes  online  time 

4  to  50  minutes  preparation  time,  and 

6  to  117  minutes  total  time 

These  general  figures  were  not  (arid  cannot  be)  used  for_ 
correlation  between  efficiency  arid  effectiveness  measures.     In  the 
statistical  analysis  actual  figures  for  each  question  and  search  were 
used*    Correlations  were  done  and  conclusions  were  derived  on  a 
search  by  search  basis. 

The  figures  presierited  here  provide  only  a  general  overview  of 
the  efficiency  characteristics  of  searches  in  this  study. 
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Table  10=^1 

EFFICIENCY  MEASURES  FOR  OUTSIDE  SEARCHES  (N  searches  »  200) 


Variable 

Minimum 

Maximum 

Mean 

^Candard 
Deviation 

SVI 

3 

45 

15.725 

8.1277 

SVJ 

1 

14 

3.73 

2.2544 

SVK 

1 

61 

10.22 

7.0931 

SVK 

0.036 

t.24d 

0.2641 

0.1792 

SVM 

0.033 

0.833 

0.2500 

0.1576 

SVN 

0.1540 

1.9560 

0.5141 

0.2929 

SVI  -  Number  of  Commands  Used  in  a  Search_ 
SVJ  «  Nimber  of  Command  Cycles  Used  in  a  Search 
SVK  »  Number  of  Search  Terms  Used  in  a  Search 
SVt  "  Online  Connect  Time  Used  for  a  Search 
SVM  =  Preparation  Time  Used  for  a  Search 
SVN  -  Total  Time  Used  for  a  Search 


EKLC 
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Table  10-2 


EFFieiENGY  fffiASURES  FOR  PROJECT  SEARCHES  (N  serrches  »  160) 


  ...  ...  ..  Standard 

Variable       MitiiffiUffi         Maximum         Mean  Deviation 


SVI  2  50  12.9937  6.9368 

SVJ  i  8  2.9812  1.5025 

SVK                 1  49  10.4687  6.9688 

SVL  0.006  0.8110  0.2046  0*1192 

SVM  0.P16  0.5670  0.1773  0.1093 

SVN  0.1050  0.8940  0.3820  0.1712 


Syi  »  Number  of  Cdnmahds  Used  in  a  Search 
SVJ  »  Number  of  Command  Cycles  Used  in  a  Search 
SVK  -  Number  of  Search  Terms  Used  in  a  Search 
£JVL  «  Online  Connect  Time  Used  for  a  Search 
SVM  *  Preparation  Time  Used  for  a  Search 
SVN  -  Total  Time  Used  for  a  Search 


ERIC 
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Table  10-3 


EFFICIENCY  MEASURES  FOR  ALL  SEARCHES  (N  searches  »  360) 


Standard 

Variable       llinimam         Maxiirt  Mean  Deviation 


SVI 

2  50 

14.511 

7.732 

SVJ 

i  14 

3.397 

1.989 

SVK 

i  6i 

10.331 

7.030 

svt 

0.006  1.24b 

0.238 

0.158 

SVM 

0.016  0.833 

0.218 

0.143 

SVN 

0.1050  1.9560 

0.455 

0.255 

SVI 

an 

Number  of  Commands  Used  iii  a 

Search 

SVJ 

a* 

Number  of  Command  Cycles  Used  in  a  Search 

SVK 

Number  of  Search  Terms  Used 

in  a  Search 

svt 

Online  Cpririect  Time  Used  for 

a  Search 

SVM 

3S 

Preparation  Time  Used  for  a 

Search 

SVN 

m 

Total  Time  Used  for  a  Search 

ERIC 
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11.  ITEMS  RETRIEVED 


This  chapter  contains  the  sunnnary  of  Items  retrieved  for  the  40 
questions  used  in  the  study i    The  items  are  summed  tip  in  two  ways: 

1»  Sum  of  all  itemajretrieved-  oyer  all-Searches :  number  of  items 
retrieved  by  each  search  suimned  over  all  searches  for  40 
questions i    Dupiicatc;S  of  identical  items  retrieved  for  the 
same  question  were  4aot^  eliminated  in  this  sxm. 

2i  Sum  of  ^ique  items  over^Xl^aearches ;  this  involves  the 
number  of  items  retrieved  aft^r  duplicates  were  eliminated. 

™??^^9^®d>  for  each  question  a  union  of  items  retrieved 
was  created  by  merging  the  output  from  all  nine  searches  and 
eliminating  duplicates.    This  simi  of  unique  items  consists  of 
the  sum  of  these  unions  for  40  questions. 

 The  difference  between  the  two  stms  is  the  same  as  the 

difference  between  tokens  and  types  in  word  counts.    To  provide  an 
example,  let's  say  that  in  searching  the  same  question  one  search 
retrieved  30  items  and  the  second  search  provided  35  items i    When  the 
two  sets  resulting  respectively  from  the  two  searches  are  compared, 
10  items  are  found  to  be  identical  between  the  ty/Oi    We  merged  the 
two  sets  of  retrieved  items  into  a  union  set  and  eliminated 
duplicates.    While  the  total  number  of  items  retrieved  (sum  over 
searches)  for  the  two  searches  is  65,  the  union  (sura  of  unique  items) 
is  55  (10- of  the  same  items  retrieved  by  both  searches  plus  20  items 
unique  to  the  first  search  plus  2^  iteias  unique  to  the  second 
search). 

All  items  retrieved  were  tared  to  analyze  the  variables  connected 
with  searches  and  searchers  and    re  'jnique  iteii*s  retrieved  were  used 
to  axialyze  variables  connected  wit!',  th^  queBti(>n. 

Table  11 provides  the  sum  of  ull  items  retrieved  by  all 
searches  before  elimination  of  duplicates  and  the  sum  of  unique  items 
retrieved  after  elimination  of  duplicates.    As  mehtioried^  if  a  search 
exceeded  150  items,  only  the  first  150  were  setit  to  the  users  for 
®y?^???^^"^r  table  shows  the  niomber  of  it^ms  evaluated  and  not 

evaluted  by  the  users,  and  total  retrievef.  lr.ams_fbr  all  350 
searchers  for  40  questions.    Only ^he  Evaluated  items  were  u^.  d  for 
statistical  analysis;  thus,  with  the  exception  of  Table  11-4  where 
all  retrieval  data  is  given  on  a  question  by  question  basis,  the 
number  of  retrieved  but  not  evaltiated  items  and  the  total  retrieval 
(evaluated  r.lus  not  evaluated  items)  is  not  treated  any  place  else^ 

Table^ll^  breaks  down  the  sum  of  all  items  retrieved  (before 
elimination  of  duplicates)  for  all  searches  by  (1)  outside  searches, 
(11)  project  searches^  and  (ill)  total  searches  and  provides  the 
number  of  items  judged  by  users  as  relevant^  partially  relevant  and 
not  relevant.    These  were  used  to  establish  a  benchmark  for  analysis 
of  variables  related  to  the  search  and  to  searching  (variables  2  to 
15  and  82  to  89  In  Table  7-1). 
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Table  11-3  breaks  down  the  sum  of  S^e  unions  of  unique  It^s 
retrieved  (after  elimination  of  aupllcate^O  for  all  s©  cches  and  for 
outside  searches  by  the  same  categories  as  above  and  V*^     .2  relevance 
evaluations  by  users.    These  were  used  to  establish  a  bencliinark  for 
analysis  of  variables  related  to  questions  (variables  16  to  28,  and 
52  to  59). 

As  can  Be  seen: 

the  360  searches  for  40  questions  retrieved  all  together 
17,695  items,  of  these,  11,796  items  were  unique  and  5899  items 
were  retrieved  by  more  than  one  search,  i.e.,  the  overlap  was 
about  33^. 

.i.  Of  the  17,695  retrieved  items  by  all  searches,  8956  (or  51%) 
were  evaluated,  the  rest  were  not.    The  fipt  evaluated  items; 
belong  to  tbe  set  of  items  exceeding  150  per  question.  These? 
8956  evaluated  items  were  used  in  the  analysiia  of  search  am 
searcher.    The  not  evaluated  items  were  not  used  In  any 
analysis . 

...  Of  the  11,796  unique  items  retrieved,  5411  (of  46%)  were 
evaluated  by  users.    These  5411.  items  were  used  in  analysis  of 
variables  related  to  questions. 

...  To  recapitulate  I  the  siim  of  Items  retrieved  (counting  search 
by  search  for  360  searches)  before  elimination  of  duplicates 
consisted  of  8956  evaluated  items  and  after  eiimination  of 
duplicates,  the  total  number  of  evaluated  items  was  5411 
(counting  question  by  question  the  items  sent  to  users).  This 
means  that  3545  (89i6  minus  5411)  items  were  retrieved  more  than 
once  and  the  overlap  In  retrieral  of  evaluated  items  was  about 
40%. 

...  Of  the  8956  items  retrieved  by  all  searches  (before 
elimihatibn  of  duplicates),  about  54%  were  the  result  of  200 
outside  searches  and  46%  the  result  of  160  project  searches. 

•.i  Of  8956  items  retrieved  by  all  searches,  the  users  judged 
approximately  31%  as  relevant,  2^%  as  partially  relevant,  and 
4i%  as  not  relevant i 

iii  Of  the  5411  unique  items  retrieved,  about  25%  were  judged 
relevant.^  27%  partially  relevant,  and  48%  not  relevant. 


 Table  ll^^i  provides  the  retrieval  data  on  a  questioxi  by  question 

basis.  The  designation  for  variables  in  the  table  are  taken  fro:?  the 
code  book  in  Table  7-1^    The  table  includes: 

1.  Question  number 

2.  File  number  of  the  DIALOG  database  used  to  search  the 
question 
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3.  QVA  -  Number  of  relevant  items  retrieved  for  a  question 

4*  QVB  »  Number  of  partially  relevant  items  retrieved  for  a 
question 

5.  RpR  »  Sum  of  the  number  of  relevant  and  partially  relevant 

items  retrieved  for  a  question 

6.  QVE  «  Number  of  not  relevant  items  retrieved  for  a  question 

7w  QVD        otal  number  of  evaluted  items  retrieved  for  a  question 

8.  QVE  »  Number  of  not  evaluated  items  retrieved  for  a  question 

9i  QVF  «  Total  number  of  items  retrieved  for  a  question 

Table is  devoted  to  ranges,  means  and  standard  deviations 
of  retrieved  and  evaluated  items  by  different  sets  of  searches: 

1.  for  all  items  retrieved  (before  elimination  of  duplicates) 
for  al5  searches  (8956  Items  retrieved  for  360  searches) 

2.  for  all  items  retrieved  by  outside  searches  (4841  items 
retrieved  for  200  searches) 

3.  for  a^li  items  retrieved  by  project  searches  (4115  items  for 
160  sear'^hes) 

T^hle  11-6  shows  ranges*  means  anr*  standard  deviation  for  unique 
items  retrieved  (after  elimination  of  duplicates)  for  all  searches 
(5411  items  for  360  searches). 

A  range*  mean*  and  standard  deviation  are  provided  for  the 
following  data  in  the  two  tables: 

1.  SVA  »  Number  of  relevant  items  retrieved  by  a  search  of  a 

question 

2.  SVB  »  Number  of  partially  relevant  items  retrieved  by  a 

search  of  a  question 

3  m  SVe  =■  relevant  items  retrieved  by  a  search  of  a 
question 

4.  SVD  «  Total  nxxmber  of  items  retrieved  by  a  search  of  a 

question  and  evaluated  by  the  user 

5.  SVE  "  Number  of  items  retrieved  by  a  search  of  a  question  but 

not  evaluated  by  the  user 

6.  SVF  "  Total  number  of  items  retrieved  by  a  search  of  a 

question 
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From  how  oh  we  are  providihg  the  legend  for  variables  under 
discussion  ih  the  related  tables  only.    We  provided  them  in  the  text 
in  this  chapter  to  give  an  example  of  how  the  data  are  described  in 
the  tables  and  chapters  that  follow. 

As  can  be  seen^  the  range  of  all  values  varies  a  lot*  The 
average  number  of  items  retrieved  for  ail  searches  is  about  49,  of 
which  about  50%  were  evaluated  and  50%  were  not  evaluated i  The 
average  number  of  relevant  itens  for  ail  searches  was  about  8, 
partially  relevant  about  7,  and  not  relevant  about  16 i    The  means 
were  somewhat  higher  for  outside  searches  than  they  were  for  project 
searches,  but  the  ranges  for  outside  searches  were  much  wider. 

  When  considering  questions  (i.e.,  the  number  of  unique  items 

retrieved  for  a  question  when  duplicates  were  eliminated),  the  ranges 
were  also  wide.    The  mean  number  for  all  unique  items  retrieved  per 
question  was  295  of  which  about  135  (46%)  were  evaluated.  Among 
items  evaluated,  means  of  the  judgements  were  about  34  relevant,  36 
partially  relevant,  and  65  not  relevant  items  per  question. 
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Table  11^3. 


TOTAL  NUMBER  OF  EVALUATED  AND  NOT  EVALUATED  ITEMS  RETRIEVED  BY  ALL 
SEARCHES  (N  Searches  «  360) 


Not 

Evaluated 

Evaluated 

Total 

All  Items 

Including  duplicates 

8956 

8739 

17,695 

All  unique  items 

<2,j(ciudlng  duplicates 

5411 

6385 

ii,796 
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USER  RELEVANCE  JUDGEMENT^ON  ALL  ITEMS  RETRIEVED  AND  EVALUATED  .(iScldaing 
duplicates)  (N  items.  «  8956;  N  all  searches  "  360;  N  outside  searches  = 
200j  N  project  searches  »  160) 


Retrieved  by  Retrieved  by 

Outside  Project  All 

User   Searches  Searches  Searches 

Judgement  Noi         %  Noi         %  No.  % 

Relevant  1378        15.A  1371        15.3  2749  30.7 

Partially  Relevant        1326        14.8  1212        13.5  2538  28.3 

R  +  pR  2704       30.2  2583        28.8  5287  59.0 

Not  Relevant  2137        23.9  1532        17.1  3669  41.0 

TOTAL  4841        54.1  4115        45.9  8956  100 
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USER  RELEVANCE  JUDGEMENTS  ON  UNIQUE  ITEMS  RETRIEVED  (excluding 
duplicateis)  FOR  ALL  SEARCHES  AND  FOR  OUTSIDE  SEARCHES  (N  unique  items  « 
5411;  N  all  searches  »  360;  N  outside  searches  »  200) 

The  number  of  unique  (or  distinct)  items  retrieved  by  outside  searches 
was  used  in  several  analyses,  but  the  number  of  unique  items  retrieved 
for  project  searches  was  not  used,  so  it  is  not  reported  here.  An 
additional  reason  for  hot  reporting  it  is  to  avoid  cohfusibh.  Unique 
items  for  outside  searches  plus  unique  itemsfor  project  searches  add  up 
to  more  than  5411  (total  unique)^  because  5411  is  a  union  and  hot  a  sum 
of  the  two  values.     (E.g.,  the  same  item  could  be  unique  in  the  set  of 
outside  searches  and  unique  in  the  set  of  project  searches ^  yet  it  is 
still  only  one  item  and  when  the  two  sets  are  merged  it  will  be  counted 
as  one  item  not  two.) 


Retrieved  by  Retrieved  by 

Outside  All 

User  Searches  Searches 

Judgement  No.  %  No.  % 

Relevant  924         25.1  1343  24.8 

Partially  Relevant        972         26.3  1448  26.7 

R  +  pR  1896        51.4  2791  51.5 

Not  Relevant  1793        48.6  2620  48.5 

TOTAL  3689        100  5411  100 
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fable  11-4 


REmiEVEb  AND  EVAtUATED  ITEMS  FOR  EACH  QUESTION  (N  questlonis  =  40) 


DIALOG 
Question  File 


lYti  1^  13  i" 

OVA 

X  V  jD 

nvn 

V  Ci 

nvi? 

001 

Oil 

27 

4n 

73 

1  HO 

774 

002 

218 

37 

oo 

73 

n 

99Q 

003 

064 

47 

8*^ 

979 

49^ 

004 

fiO 

S8 

1  1  8 

1  51 

Q 
O 

00^ 

X  \J 

23 

3Q 

n 

87 

006 

006 

X  X 

16 

134 

1  50 
X  w 

88 

_  vo 

938 

007 

075 

70 

3Q 

1  HQ 

40 

1 4g 

X  t  7 

335 

484 

008 

154 

2 

q 

7 

_  / 

54 

\J  X 

n 

61 
Ox 

009 

037 

18 

X  \J 

48 

66 

84 

1  50 

4Q5 

•+7  J 

645 

010 

154 

X 

A 

1  5 

1  Q 

14Q 

X  H  7 

310 

^  X<J 

450 

H  J7 

01 1 

154 

g 

25 

34 

X  x^ 

14Q 

X  H  7 

161 

XQ  X 

310 

OXQ 

012 

013 

6 

21 

A  X 

27 

121 

X  ^  X 

148 

X  HO 

QO 

7U 

238 

013 

015 

7 

36 

43 

106 

X  \J\J 

14Q 

X*t7 

14 

X*T 

163 

014 

151 

35 

71 

106 

51 

J  X 

157 

X  J  / 

9 

1 

X  J7 

015 

154 

28 

86 

114 

36 

1 50 

X 

l79 

322 

016 

Oil 

37 

62 

1  08 

170 

0 

1 7h 

017 

005 

36 

26 

62 

88 

150 

377 

527 

Old 

015 

66 

38 

104 

46 

150 

612 

762 

019 

075 

27 

49 

76 

74 

150 

52 

202 

020 

015 

26 

43 

69 

81 

150 

161 

311 

021 

037 

19 

6 

25 

77 

102 

0 

102 

022 

108 

is 

135 

ISO 

0 

150 

365 

515 

023 

032 

29 

9 

38 

51 

89 

0 

89 

024 

191 

i 

4 

5 

30 

35 

0 

35 

025 

001 

3b 

26 

56 

94 

150 

430 

580 

026 

038 

37 

39 

76 

8 

84 

0 

84 

027 

013 

35 

69 

014 

46 

150 

58 

208 

028 

038 

5 

23 

28 

39 

67 

0 

67 

029 

008 

36 

34 

70 

80 

150 

77 

227 

030 

071 

57 

25 

82 

13 

95 

0 

95 

Continued  on  next  page 
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Table  11-4  cont* 


Question 

File 

Number 

 _. 

Searched 



QVA 

QVB 

  _ 

RpR 

,  

QVC 

QVD 

QVE 

QVi? 



631 

061 

14 

15 

29 

85 

114 

0 

114 

032 

008 

113 

19 

132 

18 

150 

298 

448 

033 

008 

44 

57 

101 

49 

150 

123 

273 

034 

013 

lb 

39 

49 

100 

149 

207 

356 

035 

154 

31 

20 

51 

14 

65 

0 

65 

036 

090 

62 

49 

111 

39 

150 

57 

207 

037 

016 

78 

18 

96 

54 

150 

69 

219 

038 

061 

79 

29 

108 

42 

150 

595 

745 

039 

015 

26 

38 

64 

102 

166 

0 

166 

040 

016 

77 

40 

117 

32 

149 

331 

480 

1343 

1448 

2791 

2620 

5411  . 

6385 

11,796 

QVA  «  Total  Number  of  Relevant  Items  Retrieved  for  a  Question 
QVB  a  Total  Number  of  Partially  Relevant  Items  Retrieved  for  a  Question 
RpR  »  Total  Number  of  Relevant  Pius  Pari tally  Relevant  items  Retrieved 
for  a  Question 

QVG  «  Total  Nimber  of  Not  Relevant  Items  Retrieved  for  a  Question 
QVD  «  Total  NcSber  of  Evaluated  Items  Retrieved  for  a  Question 
QVE  «  Total  Number  of  Not  Evaluated  Items  Retrieved  for  a  Question 
QVF  a  Total  Number  of  items  Retrieved  for  a  Question 
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Table  11-5 


RANGE  OF  VALUE  Si  MEANj  AND  STANDARD  DEVIATION  FOR  ALL  ITEMS  RETRIEVED  .. 
(inciuding  duplicates)  BY  ALL  SEARCHES  (N  items  =»  8956;  N  all  searches  » 
360;  N  outside  searches  =  200;  N  project  searches  ■»  160) 


ALL 

SEARCHES 


OUTSIDE 
SEARCHES 


Variable    Minimum     Maximum  Mean 


SVA 
SVB 

sve 
svb 

SVE 
SVF 


0 

d 
d 
d 
d 
d 


86 

ii3 

89 
125 
54i 

656 


7.636 
7;  050 
ld;192 
24;878 
24.311 
49.189 


Variable    Minimum-     Maximum  Mean 


SVA 
SVB 
SVC 
SVD 
SVE 
SVF 


0 
0 
0 
0 
0 
0 


58 
61 

70 

118 
541 
656 


6.890 
6.630 
10.684 
24.204 
25.035 
49.239 


Standard 
Deviation 


Id. 749 
10.917 
13;886 
24.535 
54.498 
70.325 


Standard 
Deviation 


9.708 
9.787 
13.762 
23.900 
62.664 
77.346 


Standard 


Variable 

Minimum 

Maximum 

Mean 

Deviation 

PROJECT 

SVA 

0 

86 

8i568 

11.887 

SEARCHES 

SVB 

0 

113 

7.575 

12.195 

SVC 

0 

89 

9.575 

I4.d57 

SVD 

0 

125 

25.718 

25.357 

SVE 

0 

2ld 

23.4d6 

42.306 

SVF 

1 

294 

49;125 

60;654 

SVA  ■  ::Sainber  of  Relevant  Items  Retrieved  by  a  Search 
SVB  «  Number  of  Partially  Relevant  Items  Retrieved  by  a  Search 
SVG  =■  Number  of  Not  Relevant  Items  Retrieved  by  a  Search 
SVD  »  Total  Number  of  Items  Retrieved  by  a  Search  and  Evaluated  by  the 
User 

SVE  «  Items  Retrieved  by  a  Search  but  Not  Evaluated  by  the  User 
SVF  «  Total  Number  of  Items  Retrieved  by  a  Search  of  a  Question 
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Table  J  t-t 


RANGE  GF  VALUES,  MEAN,  AND  STANDARD  DEVIATION  FOR  UNIQUE  ITEMS  RETRIEVED 
(excluding  duplicates)  BY  ALL  SEARCHES  (N  items  -5411;  N  searches  »  360) 


Standard 


Variable 

Minimum 

Maxlmtim 

Mean 

Deviation 

QVA 

1 

113 

33.575 

25.322 

QVB 

4 

135 

36.200 

24.389 

QVC 

0 

156 

65.500 

37.325 

QVD 

35 

229 

135.275 

36.271 

QVE 

0 

626 

159.825 

191.330 

QVF 

35 

774 

295.100 

206.135 

QVA  »  Total  Number  of  Relevant  Items  Retrieved  for  a  Question 

QVB  »  Total  Ntmber  of  Partially  Relevant  Items  Retrieved  for  a  Question 

QVe  »  Total  Number  of  Not  Relevant  Items  Retrieved  for  a  Question 

QVD  »  Total  Number  of  Evaluated  Items  Retrieved  for  a  Question 

QVE  »  Total  Number  of  Not  Evaluated  Items  Retrieved  for  a  Question 

QVF  »  Total  Number  of  Items  Retrieved  for  a  Question 
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12.  EFFECTIVENESS  AND  OVEBUiAP  OF  SEARCHES 


^eeffectlvehass  measures  are  based  oh  two  criteria:  relevance 
and  utility.    Users  judged  the  relevance  of  each  item  retrieved  and 
the  utility  of  the  search  output  as  a  whole,    the  relevance  based 
measures  are  recall  and  precision  (Section  5.5.3)  and  the  five 
utility  based  measures  are:  overall  worth,  dollar  valui,  time  spent 
on  evaluation,  contribution  to  problem  resolution  and  satisfaction 
(Section  5.5.4  and  Table  12-4  below). ,  Recall  and  precision  pertain 
to  individual  Items  in  a  search,  and  utility  measures  to  the 
aggregate  of  all  items  submitted  to  the  user  in  response  to  his/her 
question. 


12.1  Recall  and  Precision 


 Recall  was  established  here  as  a  fraction  of  relevant  or 

partially  relevant  items  in  a  search  (R  +  pR)  in  relation  to  all 
^_"^_P^_  i^®™^  ^^^^^  of  ^11  nine  searches  for  a  question. 

Precision  was  measured  iti  two  ways: 

1-  Precisi^Ma^fo^^  a  search  (SVH)  -  fraction  of  relevant  items  in 
a  given  search  in  relation  to  all  items  retrieved  by  that 
search 

2.  Precision  for  a  guestioh  (QVH)  -  fraction  of  relevant  items 
ih_ relation  to  all  items  submitted  to  the  user,  i.e.,  in 
relation  to  the  union  output  of  the  nine  searches  for  a 
question 

In  other  words,  precision  measures:  (i)  the  effectiveness  of  the 
retrieval  of  each  search  for  a  question,  and  (ii)  the  effectiveness 
of  retrieval  of  all  searches  tbgetheri    Recall  measures  the 
effectiveness  of  retfle-^ral  of  each  search  for  a  question  only. 
Recall  for  all  searches  for  a  question  (iiei,  for  the  union  of 
retrieval  from  nine  searches)  carmot  be  established  becasue  we  do  not 
know  what  other  relevant  items  were  left  unretrleved  in  the  file. 
While  precision  is  an  absolute  measure,  recall  is  a  comparative 
measure  among  the  nine  searches  for  a  question. 

Table  12-1  provides  the  range,  mean  and  standard  deviation  of 
recall  (SVG)  and  precision  (SVH)  for  all  searches  for  the  40 
questions.    Tabie-12^2  provides  the  range  arid  mean  of  recall  (SVG) 
and  precision  (SVH)  for  the  nine  seat:ches  for  each  question,  on  a 
question  by  question  basis. 

Table  12-^3  provides  the  range,  mean  and  standard  deviation  of 
overall  precision  (QVH)  for  all  questions  together  with  the  liit  of 
question  ntnnbers  falling  into  a  given,  range.     This  list  could  be 
viewed  as  a  histogram  identifying  peaks  and  valleys  at  different 
ranges « 
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As  metitlbned  before ,  the  mean  values  presented  have  to  be 
Interpreted  with  caution,  becauise  the  distributions  are  not  normal 
(bell  shaped  with  one  peak).    As  can  be  seen  from  the  histogram  in 
Table  12-3  >  there  are  several  different  peaks  and  the  mean  precision 
of  51%  falls  at  one  of  the  least  represented  mean  value 

is  like  saying  that  the  popui a ^  the  U;S*  is 

someplace  in  the  West  where  in  fact  very  few  people  live,  or  like 
saying  when  one  person  ate  a  meal  and  the  other  did  not  that  on  the 
average  each  had  half  a  meal i    We  have  been  reluctant  to  provide 
means  because  they  can  be  easily  misinterpreted:  we  are  providing 
them  f^y^ay  because  of  expectations.    We  are  also,  therefore, 
including  these  notes  of  caution. 

As  can  be  seen: 

...  the  average  recall  for  all  searches  was  about  22%  and  the 
average  precision  about  57% 

...  the  outside  searches  did  somewhat  poorer  on  recall  and 
better  on  precision  than  the  project  searches 

...  the  ranges  of  precision  arid  recall  for  the  nine  searches  for 
a  question  varied  widely  for  almost  all  questions;  there  were 
very  few  questions  with  a  narrow  range.    The  searches  for  the 
same  question  differed  considerably  in  effectiveness  measures . 

Figure  12-1  provides  a  plot  of  recall  versus  precision  for  all 
360  searches.    For  each  search^  recall  and.  precision  are  plotted 
against  each  other  as  one  points  resulting  in  360  points  plotted  in 
the  graph.    Two  linear  regression  lines  are  plotted:  Is  for  precision 
as  ah  independent  variable  and  recall  as  the  dependent  variable 
(connect  Y  to  Y  on  the  vertical  lines),  and  2i  for  recall  as  an 
Independent  variable  and  precision  as  the  dependent  variable  (connect 
X  to  X  on  the  horizontal  lines). 

The  results  are  Q^ite  remarkable i    An  enormous  amount  of  scatter 
is  showni    it  is  often  said  that  there  is  an  inverse  relation  between 
recall  and  precision.    In  real  search  situations  it  has  never  been 
clear  that  this  relationship  should  exist,  particularly  in  cases 
where  the  size  of  the  retrieved  set  is  limited  by  economic  factors. 
With  our  data,  no  matter  which  variable  is  chosen  as  independent, 
there  is  a  positive  relationship  between  recall  and  precision:  as  biie 
rises  so  does  the  other.    This  directly  contradicts  many  established 
results  including  the  Cleverdon  Law  on  the  inverse  relationship 
between  recall  arid  precision. 

We  did  riot  study  why  the  observed  relationship  between  recall 
arid  precisibri  bccurred  iri  our  data.    We  have  only  observed  what  the 
data  from  the_8tudy  showed  the  relationship  was  for  360  searches 
retrieving  8,956  items  for  40  questious.    We  can  only  speculate, 
albrig  with  everyone  else,  why  our  observation  flies  in  the  face  of 
others . 
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12.2  Utility  MeastigBS^ 


The  text  and  scales  fot  che  five  utility  measures  are  given  in 
lafele  12^,  along  with  the  number  of  users  assigning  the  given  value 
dtl  the  scale. 

Table  12-5  is  a  comparison  table  providing  further  analysis  of 
the  data,  that  is  the  range i  mean  and  standard  deviation  of  the 
utility  measures  assigned  by  the  users* 

_  Tables  12-6  to  12'"10  are  eacfi^  in  turn^  devoted  to  the  five 
utility  measures  which  provided,  for  each  ^7^aiue  on  the  given  scale, 
the  number  of  questions  for  which  the  value  was  assigned  and  the 
actual  question  numbers  involved i    Table  12-^  is  devoted  to  time 
spent  in  evaluation;  Table  12-7  to  dollar  value  assigned;  Table  12-8 
to  worth  of  a  search;  Table  J; 2-9^  to  contribution  to  problem 
resolution;  and  Table  12^15  to  overall  satisfaction. 

As  can  be  seen: 

••i  70%  of  the  users  considered  their  participation  in  the 
project  and  the  information  that  resulted  as  worth  "much  more" 
or  "somewhat  more"  than  the  time  it  took;  20%  said  it  was  worth 
"about  the  same"  as  the  time  it  took,  and  10%  said  it  was  worth 
"less"  than  the  time  it  took 

45%  of  the  users  could  hot  assign  a  dollar _yalue  to  the 
information  provided;  28%  assigned  less  than  $50;  20%  assigned 
between  $50  and  $200;  and  7%  assigned  over  $200 

. . •  25%  of  the  users  spent  less  than  a  half  hour  on  evaluation; 
30%  spent  between  one  and  two  hours;  and  45%  spent  more  than  two 
hours 

• • •  about  48%  of  the  users  scored  the  contribution  made  by  the 
information  supplied  to  the  resolution  of  their  probietn  as  high 
(upper  two  points);  20%  were  in  the  middle  and  32%  were  in  the 
lower  two  points  on  the  scale.    Actually,  only  two  users  (5%) 
said  "nothing"  was  contributed. 

...  58%  of  the  users  scored  their  satisfaction  with  the  results 
of  the  search  high  (upper  two  points);  22%  scored  in  the  middle; 
and  20%  scored  on  the  lowest  two  points.    Actually,  only  two 
users  (5%)  said  they  were  "dissatisfied". 

It  is  of  interest  to  note  that  the  problem  resolution  scores  did 
not  parallel  the  satisfaction  scores.    Six  (10%)  more  users  Jcbred 
satisfaction  high  than  scored  problem  resolution  high>  and  five  (12%) 
^9T^  scored  problem  resolution  low  than  scored  satisfaction 

low.    This  shows  that  users  do  distinguish  between  the  two  concepts 
as  measures . 
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12.3  Overlap  In  Search  Terms  and  Items  Retrieved 


Two  types  of  overlap  were  studied: 

!•  Search  term;  degree  of  agriBement  Iri  selection  of  search  terns 
between  each  pair  of  searchers  searching  the  same  question 

2.  Items  retrieved;  degree  of  agreement  in  retrieval  of  items 
between  each  pair  of  searchers  searching  the  same  question* 
These  were  subdivided  into  degree  of  agreement  on  ail  items 
retrieved  (relevant ,  partially  relevant,  nonreievant)  and  on 
only  relevant  or  partially  relevant  items  retrieved 

These  measures  were  described  in  this  report  in  Sections  S«4,3 
and  5.4.4  and  will  be  discussed  further  in  Chapter  15  which  reports 
on  the  comprehensive  statistical  analyses  which  were  done.    Only  the 
basic  data  are  given  here* 

Tahl^  42^14^  provides  the  means  and  standard  deviation  for  the 
degree  of  overlap  in;  (i)  search  terms,  (ii)  all  retrieved  items 
(relevant,  partially  relevant,  arid  rionreleyant) ^  arid  (iii)  retrieved 
relevant  or  partially  releyarit  items.    Agairi,  the  actual 
distributloris,  are  skewed  (in  each  of  these  distributions  toward  zero) 
and  the  mean  has  to  be  interpreted  with  caution. 

The  number  of  comparisons  used  to  derive  the  means  and  the 
degree  of  overlap  (and  other  statistics  in  Chapter  15)  was  800.  The 
800  figure  was  established  as  follows;  for  each  question  there  were 
five  outside  searches,  each  search  waB  compared  four  times  (e.g«, 
search:  1  was  compared  with  search  2,  3,  4,  and  5  but  not  with  itself) 
and  there  were  40  questions.    Thus,  the  total  was  5  searches  X  4 
comparisons  X  46  questions  800. 

As  can  be  seen  from  the  table,  the  means  for  the  degree  of 
overlap  for  both  term  selection  and  retrieved  items  are  quite  iow^ 
Mean  agreement  on  terms  was  27%,  on  all  items  retrieved  16%,  and  on 
relevant  or  partially  relevant  items  18%.    The  means  for  the  two 
types  of  overlap,  of  search  terms  and  of  items  retrieved,  are 
<9ignif icantly  different.    Further  analysis  of  that  difference  is 
provided  in  Chapter  15^    The  searches  for  the  same  question  done  by 
different  searchers  differed  considerably  in  both  search  term 
selection  arid  iri  items  retrieved.    The  differertces  were  ever  so 
slightly  less  (2%)  when  only  relevant  or  partially  relevant  items 
were  cbrisidered. 

The  low  degree  of  overlap  amorig  searches  iri  both  term  selection 
and  items  retrieved  is  one  of  the  most  significant  observations  of 
the  study. 


87 


108 


Table  12-1 


RANGE^MEAN,  AND  STANDARD  DEVIATION  OF  RECALL  AND  PRECISION  FOR  ALL 
SEARCHES  (N  all  searches  «  360;  N  outside  searches  «  200;  N  project 
searches  ^  160) 




Measure      Minimum      Maximum     Mean  Deviation 


ALL  SVG  0  0i895         0.2193  0.213 

SEARCHES        SVH  0  1.000         0.572  0.335 


Standard 

Measure      Minimum      Maximum     Meaii  Deviation 


OUTSIDE         SVG  0  0.8080        0.1969  0.1992 

SEARCHES        SVH  0  1.0000        0.5740  0.3447 


Standard 

Measure      Minimum      Maximum     Mean  Deviation 


PROJECT         SVG  0  0*8950       0*2470  0*2272 

SEARCHES        SVH  0  1*0000        0.6120  0.3199 


SVG  -  Search  recall 
SVH  «  Search  precision 
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Table  12-2 


RECiat  AND  PRECIS iON  RANGE  AND  MEAN  FOR  THE  NINE  SEARCHES  ON  EACH 
QUESTION  (N  questions  »  40;  N  searches  per  question  ^9) 


Question  RECALL, (SVG)  _         PRECISION  (SVH) 

Number  Minimum     Maximum     Mean  Minimum     Maximum  Mean 


001 

•013 

.562 

.1322 

.286 

1,0 

.493 

002 

0*0 

.5d7 

.2313 

d.o 

.720 

i318 

A  A  O 

003 

O.Q 

.506 

.1431 

OiO 

1 .0 

.549 

A  A:^ 

005 

•  042 

•  441 

.1683 

.556 

i878 

.781 

AA  C 

005 

Q«0 

•  564 

.2448 

OiO 

i677 

i447 

006 

0.0 

•  688 

.2dl4 

Oid 

.133 

.ld6 

007 

0.0 

•  440 

.1741 

0.0 

i800 

.731 

A  A  O 

0.0 

•  857 

.4443 

OiO 

.154 

.114 

009 

•  045 

•  561 

il647 

.100 

1.0 

.44d 

A  1  A 

010 

0 .0 

•  895 

.2275 

0.0 

.6dd 

.127 

All 

Oil 

.029 

.676 

i23l7 

.225 

.5d0 

.228 

012 

i074 

i48l 

.2137 

.090 

.5dd 

.182 

A  1  ^ 

013 

i046 

i488 

.2325 

.267 

i.d 

.288 

A  1  '/ 

014 

i009 

.698 

.2398 

.030 

.841 

.675 

015 

.184 

.772 

.4853 

.746 

.886 

.76d 

016 

.080 

.661 

.2363 

.178 

.625 

.364 

017 

.016 

.5dd 

.1413 

.250 

I.d 

.413 

018 

o.d 

.8d8 

.1367 

0.0 

.744 

.693 

019 

d.d 

.472 

.2514 

0.0 

.9dd 

.5d6 

020 

.014 

.623 

.1431 

.250 

.5dd 

.46d 

021 

d.o 

.56d 

.1644 

.066 

I.d 

.245 

022 

.013 

.833 

.1924 

1.0 

I.d 

I.d 

023 

.052 

.684 

.2715 

.200 

.75d 

.426 

024 

0.0 

.6dd 

.3777 

0.0 

.333 

.142 

025 

0.0 

.554 

.14d6 

0.0 

I.d 

.373 

026 

0.0 

.711 

.1592 

0.0 

I.d 

.9d4 

027 

.019 

.587 

.2852 

.645 

.947 

.693 

028 

0.0 

.643 

.3d93 

0.0 

.818 

.417 

029 

.042 

.543 

.2332 

.359 

1.0 

.666 

030 

.012 

.793 

.3^28 

.800 

I.d 

.863 

Continued  on  next  page 
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Table  12-2  Cont. 


Question   RECALL  jSVG)  PRECISIDN  (SVH) 

Ntimber  Minimum     Maximum      Mean  Minimum     Maximum  Mean 


031 

.034 

i759 

i268i 

.050 

;349 

.393 

032 

.007 

.750 

i3d2i 

.055 

1.0 

.880 

033 

OiO 

i4d6 

ii352 

Oid 

1.0 

.673 

634 

0.0 

.327 

.1268 

.186 

i.d 

.328 

035 

.019 

i392 

.1830 

.688 

1.0 

.784 

036 

.045 

i396 

.1870 

.583 

.786 

.740 

037 

.031 

.458 

.2383 

.298 

.833 

.640 

038 

.046 

;343 

.1171 

.441 

.925 

.720 

039 

0.0 

.422 

.1301 

0.0 

.583 

.386 

040 

.025 

.427 

.1460 

.158 

1.0 

.785 

9b 

111 


RANGEj  MEAN^  AND  STANDARD  DEVIATION  OF  THE  OVERALL  PRECISION  (QVH)  FOR 
MiL  QUESTIONS  (N  questions  »  40) 

The  overall  precision  fbra  question  is  the  fraction  of  items  judged 
relevant  and  partially  relevant  by  the  user  in  relation  to  all  retrieved 
items  sent  to  the  user  for  that  question.     (Overall  recall  for  a  question 
cannot  be  calculated  because  there  was  no  knowledge  of  which  relevant 
items  were  not  retrieved  by  any  search.) 


Number  of 

Precision  Range  Questions      Question  Numbers 


.iOO  - 

.199 

5 

QO06; 

Q008; 

QOiO; 

Q012; 

Q024 

.200  - 

.299 

3 

QOll; 

Q013; 

Qb21 

.300  - 

.399 

6 

Q002; 

Qbl6; 

Q025; 

Q031; 

Q034; 

Q039 

.400  - 

.499 

7 

QOOl; 

Q005; 

Q009; 

Q017; 

Q020; 

Q023; 

.500  - 

.599 

2 

Q003; 

Q019 

.600  - 

.699 

6 

Q014; 

Q018; 

Q027; 

Q029; 

Q033; 

Q037 

.700  - 

.799 

7 

Q0045 

Q007; 

Q015; 

Q035| 

Q036; 

Q038; 

.800  - 

.899 

2 

Q030| 

Q032 

.900  - 

1.0 

2 

Q022; 

Q026 

MEAN  :  0.510 

STANDARD  DEVIATION  :  0.2439 
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Figure  12-1 


j^LDT  QF  PRECISION  AND  RECALL   (N  SEARCHES  =  360) 

Each.of  the  360  searches  is  represented  by  a  paint  in  the 
scatter  jDibt..     THe  jDrecisidh  is  plotted  dn  this  horisontal  axis 
and  the  recall    (using  the  union  of  relevant  retrieved  items  as  a 
base  -for  comparispn)   is  plotted  on  the  vertical  a^is.     A  i 
represents  a  single  point,   a  2  represents  2  points  falling  in  the 
same  place,   arid  so  bri.     Beybrid  9,,   the  letters  a,   b  ...   are  used. 
The  precision  and  recall   show  a  positive  correlation  bf  15.36%. 
The  best  regression  lirie,   whichever  is  regarded  as  the 
independent  variable,   has  a  positive  slope. 
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.33541  X=  .2493S*Y+  .51749  .10997 
.21330     Y=   .100S6*X+   .16155  .04443 


VARiABbE       9  SVH 


VERSUS  VARIABLE 


8  SVG 


SV6  ■  Search  Recall 
SVH  ■  Search  Precision 
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Table  12^4 


UTILITY  MEASURES:  DISTRIBUTION  OF  USER  ASSIGNMENTS  TO  EACH  MEASURE 
(N  users  »  40) 


WORTH  SCALE 

Was  your  participation  in  this  project  and  the  information  which  resulted: 

Number  of 
Users 


5  Worth  much  more  than  the  time  it  took  16 

4  Worth  some^^at  more  than  the  time  it  took  12 

3  Worth  about  as  much  as  the  time  it  took  8 

2  Worth  less  than  the  time  it  took  4 

i  Practically  worthless  0 


PROBLEM  RESOLUTION  SCALE 
What  contribution  has  this  infor- 
mation made  toward  the  resolution 
of  your  research  problem: 

Number  of 
Users 

5  Substantial  contribution  3 
4  16 
3  8 
2  ii 

1    Nothing  contributed  2 


SATISFACTION  SCALE 

How  satisfied  were  you  with  the 

results  of  the  search: 


Number  of 
Users 


5  Satisfied  12 

4  11 

3  9 

2  6 

1  Dissatisfied  2 


USER'S  TIME 

How  much  time  did  you  spend 
reviewing  these  items? 


Number  of 
Users 


DOLLAR  VALUE  ASSIGNED 

What  is  the  dollar  value  of 

these  items? 


Number  of 
Users 


Less  than  i  hour 
1  -  2  hours 
Over  2  hours 


19 
12 
18 


I  cannot  assign  a 

a  dollar  value  18 

Less  than  $56  11 

$56  -  $166  3 

$166  -  $200  5 

Over  $200  3' 
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Table- 12=5 


RANGE,  MEAN,  AND  STANDARD  DEVIATION  OF  UTILITY  MEASURES  ASSIGNED  BY  USERS 
(N  users  =»  40) 


Standard 

Measure         Minimum         Maximum         Mean  Deviation 


UVA  0.17  12  2.4917  2.021 

UVS  0  1000  75i25  172^475 

me  i  5  4.6  i.e 

UVD  1  5  3.156  1^663 

HVE  1  5  3.625  1,76 


UVA  =»  Time  spent  by  the  user  evaluating  the  items 
UVB  "  Dollar  value  assigned 
UVC  =  Worth  assigned 
UVb  =  Problem  resolution 
UVE  =■  Satisfaction 


94 

115 


Table  12-6 


VALUES  ASSIGNED  FOR  TIME  SPENT  BY  THE  USERS  EVALUATING  THE  ITEMS 
RETRIEVED  (UVA)  (N  users  »  40) 

"How  much  time  did  you  spend  reviewing  these  abstract?" 


Tine 

Spent 

Number  of 

Question  Number 

Less 

than  1  hour 

7 

Qbb2; 

Qb08; 

Q014; 

Qb26; 

Qb28; 

Q029; 

Qb31 

1  to 

2  hours 

12 

Q005; 
Q023; 

QOll; 
Q024; 

Q017; 
Q032; 

Q019; 
Q035; 

Q020; 
Q036 

Q021; 

Q022; 

2  to 

4  hours 

13 

Q003; 
Q025; 

Q006; 
Q030; 

Q007; 
Q033; 

Q009; 
Q034; 

Q012; 
Q037; 

Q013; 
Q039 

Q018; 

4  to 

6  hours 

7 

QOOl; 

Q004; 

QOlO; 

Q015; 

Q016; 

Q038; 

Q040 

Over 

6  hours 

1 

Q027 
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Table  12-7 


VALUES  ASSIGNED  AS  A  DOLLAR  VALUE  OF  A  SEARCH  BY  THE  USERS  (UVB) 
(N  users  »  40) 

"What  is  the  dollar  value  of  these  abstracts  to  you?" 


Number  of 

Value  Assigned  Questions      Question  Number 


I  cannot  assign 
a  dollar  value 

18 

Q002; 
Q0i2; 
Q026| 

Q004; 
Q014| 
Q027| 

Q006; 
QQ18| 
Q030; 

Q007; 
QOl  9; 
Q038 

Q008; 
Q020; 

Q009; 
Q023; 

QOll; 
Q025; 

$1  -  $50 

12 

Q003; 
Q024; 

QOlO; 
Q028; 

Q013; 
Q031; 

Qbl4; 
Q035; 

Q017; 
Q036 

Q021; 

Q022; 

$51  -  $100 

3 

Q005; 

Q016; 

Q039 

$101  -  $200 

4 

QOOl; 

Q029; 

Q032; 

Q034 

$201  -  $300 

i 

Q037 

$301  -  $400 

i 

Q040 

$1000 

i 

Qd33 
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Table  12-8 


VALUES  ASSIGNED  BY  THE  USERS  TO  THE  WORTH  OF  A  SEARCH  (UVC) 
(N  usars  =«  40) 

"Was  your  participation  in  this  project  and  the  information  which 
resulted:  .wi" 


Value 

Meaning 

Number  of 
Questions 

Question  Number 

5 

Worth  more  than  the 
time  it  took 

16 

Qdbl; 
Q020; 
Q030; 
Q038 

Qdb4; 
Q021; 
Q032; 

Qd07; 
Q023; 
Q033; 

Qdl5; 
Q026; 
Q035; 

Q019; 
Qd29; 
Q037; 

4 

Worth  somewhat  more 
than  the  time  it  took 

12 

Q002; 
Q016; 
Q039; 

Q005; 
Q018; 
Q040 

Q0G9 ; 
Q031; 

QOlOi 
q034; 

QOU; 
Q03e; 

3 

Worth  about  as  much 
as  the  time  it  took 

8 

Q003; 
Q025; 

Q008; 
Q027J 

Q012; 
Q028 

Q014; 

Q022; 

2 

Worth  less  than  the 
time  It  took 

4 

Q006; 

Q013; 

Q017; 

Q024 

1 

Practically  worthless 

0 
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Table  12-9 


VALUES  ASSIGNED  BY  THE  USERS  TG  THE  edNfRIBUTIdN  TG  THE  RESGLUtlGN  GF 
I'HEIR  PRGBLEM  3Y  THE  INF0RMATIGN  RECEIVED  (N  users  «  46) 

"What  contribution  has  this  information  made  it o  the  resolution  of  your 
problem?"    Range:  from  i  (nothing  contributed)  to  5  (substantial 
contribution) 


Number  of 

Value  Questions  Question  Numbers 


5 

3 

Q030; 

Q032; 

Q038 

4 

15 

QOOl; 
Q026; 
Q040 

goo  7; 
Q027; 

Q0115 
Q029; 

Q015J 
Q033; 

Q016; 
Q035; 

Q021J 
Q037 ; 

Q023; 
Q039; 

3 

9 

Q002; 
Q034; 

Q003; 
Q035 

Q004; 

Q005; 

Q018; 

Q019; 

Q020; 

2 

11 

Q006; 
Q022; 

Q008; 
QG25; 

Q009; 
Q028; 

QOlO; 
Q031 

Q012; 

Q013; 

Q017; 

1 

2 

Q014; 

Q024 
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Table-- 12^10 


VALUES  ASSIGNED  BY JIHE  USERS  TO  THEIR  OVERALL  SATISFACTION  WITH  THE 
RESULTS  OF  THE  SEARCH  (N  users  «  40) 


'•Hpwr  satisfied  were  you  with  the  resultis  of  the  search?"  Range  from  1 
(dissatisfied)  to  5  (satisfied) 


Number  of 

Value  Questions  Question  Numbers 


5 

11 

QOOl; 
Q030; 

Q007; 
Q032; 

QOll; 

Q037; 

QD16; 
Q038 

Q020; 

Q023; 

Q026; 

4 

13 

QO03; 
Q033; 

QO04; 
Q034; 

Q015; 
Q035; 

Q01S; 
Q036; 

Q021; 
Q039; 

Q024; 
Q04O 

Q029; 

3 

8 

Q002; 

QO06; 

Q008; 

Q009; 

QOlO; 

Q0i8; 

Q027; 

2 

6 

QbOB; 

Q013; 

Q014; 

Q017; 

Q022; 

Q025 

i 

2 

QG12; 

Qd31 
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Table  12-11 


MEAN  AND  STANDARD  DEVIATION  FOR  THE  DEGREE  OF  OVERLAP  IN  SEARCH  TERMS  AND 
ITEMS  RETRIEVED  FOR  SEARCHES  FOR  I'SE  SAME  QUESTION  {N  questions  =■  40; 
N  searches  per  qaestion  >»  5;  N  comparisons  per  search  =•  4  i±t  was  not 
compared  with  itself);  N  pairs  of  comparisons  =  800  (40  X  5  X  4)) 


Overlap 


Mean 


Standard 
Deviation 


Selection  of  search  terms 


.27 


.20 


All  Items  retrieved 


.16 


.28 


Rfclayant  oi:  i;artially 

relevant  items  retrieved 


.18 


.30 
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eSftPTER  13^  METHODS  OF  STATISTieAL  ANALYSIS 
13.1  Zhtrbduction 

 The  key  to  understanding  the  statistical  aspects  of  this 

study  of  on-line  searching  is  to  realize  that  there  are  five 
distinct  entities  involved  every  time  an  evaluation  is  made, 
Th^se  entities  are: 

The  user 

The  question 

The  seaircbef 

The  search 

The  retrieved  item 

Any  measured  variable  describes  one  or  more  6£  these 
entities.     For  example  information  about  the  intended  specificity 
or  complexity  of  the  question  describes  only  the  question*  The 
cognitive  attributes  of  the  searcher  describe  only  the  searcher. 
The  number  of  commands  or  the  number  of  search  terms  used 
describe  only  the  search.     The  data  base  identifier  and  accession 
number  of  a  retreived  item  describe  only  the  retrieved  item.  On 
the  other  hand,  the  evaluation  is  a  description  of  the  relevance 
of  the  retrieved  item,  given  by  the  user,  and  so  relates  two 
entities.    Thaoyerali^  scores  for  a  given 

question  combines  several  searches  by  several  searchers  and  so 
are  descriptive  of  the  question,  the  searches  and  the  searchers 
together • 

The  data  may  be  examined  at  each  of  several  different  levels 
of  aggregation.     Some  of  these  levels  of  aggregation  are  more 
familiar  to  the  practitioner  community,. while  others  are  more 
powerful  in  the  search. for _ pdisible  explanatory  relations. 
Corresponding  to  each  level  of  aggregation  a  specific  data  file 
may  be  formed.     It  is  formed. by  retaining  those  of  the  90 
variables  that  are  meaningful  at  that  level  of  aggrega;tibh r  and 
either  removing  or  ignoring  the  others.    With  reference  to  the 
codebobJc  of  90  variables  (Table  7-1 ) ,  we  will  briefly  describe 
the  status  of  each  of  the  variables.     In  giving  this  description 
it  is  helpful  to  think  of  a  kind  of  "backbone"  containing  five 

variables:     the  question  number^  the  database  number^  the  item  

number,  the  searcher  number^  and  the  evaluation.    This  represents 
the  finest  possible  level  of  detail  in  that  it  is  free  from  the 
confounding  influences  that  arise  in  any  real  search^     The  search 

entity  is  defined  by  giving  both  the  question  number  and  the  

searcher.     The  question  entity  is  identified  by  giving  only  the 
question.     The  range  of  applicsability  of  variables  is  as  follows; 
(as  shown  earlier  in  Table  7-1): 
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TABLE  7-1 


Variable  no  Applicability 

i  Item  

2-15  Search 

16-36  Question 

37-51  Searcher 

52-55  Question  and  User 

56-59  Question  and  Searcher 

60-81  Question 

82-90  Searcher 


The  files  used  to  investigate  various  relationships  vary 
from  files  containing  one  case  for  each  question  up  to  the  most 
complicated  file  containing  one  case  for  each. retrieved 
item.     Their  properties  are  summarized  in  Table  13-1. 


EKLC 
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Table  13-1 


OVERVIEW  OF  DATA  FILES 

Pdreach  entity  or  relation  the  correct  data  file  must  be  _ 
analyzed.  The  file  to  be  used  is  the  smallest  one  applicable. to 
all  of  the  entities  under  investigatipn.  For  example,  the  file 
with  8956  duplicated  items  contains  4115  items  occurences  of 
retrieval  by  project  searchers.  It  cannndt  be  used  to  study  the 
effect  of  searcher  characteristics  because  of  the  unequal  weight 
that  these  three  individuals  would  receive. 


Case  N  of  cases  Use  of  file 

Question  40  Characteristics  of  questions 

User  evaluation  of  unions 
Searchers  40  Cognitive  and  experience  data 

for  36  outside; 3  project  sear- 
chers; 1  judge 

Search-pro jeet  160  CharaGtexristics/comparisons 

The  4  project  searches 
Search-outside  266  The  5  outside  searches 


Overlap  800  Overlap  in  search  terms  or 

Retrieved  items  for  every 
Question  and  searcher  pair i 

Unique  item  5411  Characteristics  of  retrieved 

items^  Item-wise  analysis  of 
the  impact  of  question  charac- 
teristics on  relevance 

Distinct  item  3689  Outside  searches  only 

Duplicated  item      4841  Item-wise  analysis  of  the  impact 

of  outside  searcher 
characteristics  on  relevance. 


Duplicated  item  4115 


Duplicated  item  8956 


Item-wise  analysis  of  the  impact 
of  project  searcher 
characteristics  on  relevance. 
Item-wise  analysis  of  the  impact 
of  search  characteristics  on 
relevance. 
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The  use  of  each  file  Is  best  illustrated  by  one  or 
two  examples.     Suppose,  for  example,  we  want  to  know  what  effect 
the  amount  of  available  public  knowledge  has  on  the  precision 
and  recall  of  searches  for  a  given  question.     We  can  address  this 
directly  by  examining  the  file  with  40  cases,  one  for  each 
question.     We  can  ask  whether  the  union  of  nine  search  results  had 
high  or  low  precision  for  each  question,  and  correlate, that  with  th 
public  knowledge  variables.     On  the  other  hand,  "recall"  in  this 
experiment  has  been  defined  as  the  fraction  of_all  relevant 
retrieved  items  retrieved  in  a  given  search.     Therefore,  to 
examine  the  relation. between  recall  and  a  search  characteristic 
we  must  go  to  the  files  with  160  cases  (for  the  four  project 
searches)  and  with  200  cases  (for  the  outside  searches). 

If  we  want  to  know  the  relative  effectiveness  of  the  four 
types  of  project  searches  we  will  address  the  question  to  the 
file  with  160  cases.     If  we  want  to  know  the  impact  of  the 
cognitive  characteristics  of  the  searcher  oh  precision  and 
recall  we  will  address  the  question  to  the  file  with  260  cases. 


If  we  want  to  relate  characteristics  of  the  retrieved  items 
to  characteristics  of  the  question,  without  regard  to  the  search  or 
searcher  who  retrieved  the  items  we  address  our  questions  to  the 
file  with  5,411  cases. 

Finally,  if  we  want  to  keep  track  of  the  retrieved  items  and 
the  detailed  characteristics  of  the  search  that  retrieved  them  we 
may  address  the  question  to  the  file  with  13,956  cases. 

The  difference,  13,956  minus  5,411  or  3,545  represents  the 
number  of  times  an  item  was  retrieved  more  than  once.  In 
order  to  study  the  effects  of  multiple  retrieval  on  the  expected 
relevance  of  an  item  we  have  added  one  variable  to  the  file 
with  5^411  cases.     This  isthe  number  of  times  that  each 
item  is  retrieved  by  outside  searchers,  and  can  take  the 
values  1-5. 


13.2  Significance  and  In^rtance 

It  is  important  to  distinguish  between  findings  that  are 
statistically  significant  and  those  findings  that  provide  a 
substantial  explanation  of  the  relevance  of  retrieved  items. 


When  a  statistically  significant  relationship  is  found  it 
can  be  assigned  some  measure  of  association.     This  is  a  imeasure 
of  the  extent  to  which  one  of  the  variables  in  question  (the  one 
presumed  independent)  determines  the  other  (the  one  taken  to  be 
dependent).     The  fact  that  a  relationship  is  significant  does 
hot  mean  that  it  is  in^ortaht. 
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We  regard  a  relationship  as  inpprtant  if  the  independent 
variable  explains  a  substantial  amount  of  the  observed  variation 
in  the  dependent  variable.    The  measures  of  importance  that  have 
been  used  in  this  study  are  the  R-^squared  measure  for  regression 
analysis,  and  the  t  value  for  analysis  of  the  log  cross  ratio. 
We  pause  for  a  moment  to  review  the  meanings  of  these  methods. 

13*3  Regression  analysis 

in  a  regression  analysis,  exploring  the  dependence  of  Y  on 
the  analyst  (in  thiscase^  aided  by  the  BMDP  packaged  tries  to 
find  the  best  straight,  line  describing  7  as  a  function  of  X.  We 
may  imagine  all  the  values  of  Y  and  X  plotted  in  a  single  graph. 
The  calculations  themselves  have  been  perf romed  using  the  BMDP 
package  for  this  and  all  of  the  other  statistical  analyses 
described  in  Chapters  13-16 ^  113-1] 

When  such  a  graph  is  made  the  values  of  Y  will  show  some 
substantial  variation.     This  degree  of  variation  is  conveniently 
summarized  by  a  statistical  quantity  called  the  variance.  The 
variance  is  the  average  value  of  the  square  of  the  difference 
between  any  particular  value  of  Y  and  the  average  of  all  the 
values  of  Y^    When  a  line  is  fit  to  the  data,  to  explain  Y, 
a  certain  amount  of  the  value  of  Y  remains  unexplained.  The 
average  square  of  the  unexplained  part  is  called  the  residual 
mean  square  variation.     The  difference  between  the  two  is  the 
part  of  the  variance  that  is  explained  by  the  model.     This  may  be 
expressed  as  a  percentage  of  the  original  variance,  which  is 
called  R-sguared.     Thus,  if  R- squared  is  8        the  model  explains 
80%  of  the  original  observed  variation  in  the  values  of  Y.  If 
R- squared  is  10%,  90%  of  the  original  variation  means 
unexplained.    As  a  rough  rule  of  thumb,  models  that  explain  less 
than  50%  of  the  variance  will  not  be  regarded  as  important  even 
if  they  are  statistically  significant. 

13.4  Logarithmic  Cross  Ratio  Analysis 

in  analysis  of  the  large  files^  containing  one  entry  for 
each  retrieved  item,  we_have  used  a  powerful  technique  called 
cross  ratio  analysis.     To  apply  the  cross  ratio  analysis^  each 
variable. (for  which  such  a  distihctioh  is  meaningful)  is  broken 
into  a  class  of  high  values  ahd_ a  class  of  low  values.  For 
cohvehiehce  the  mean  is  generally  taken  as  the  dividing  line. 
Since  the  dependent  variable  of  greatest  interest  is  the 
relevance  of  retrieved  item  we  take  for  high  value  of  the 
relevance*^  "relevant "  or  "partially  relevaht"^     Evesry  case  may  then 
be  classified  into  exactly  one  of  four  cells  in  a  2  x  2  table. 
The  number  of  cases  for  which  the  variable  is  low  and  the  item  is 
hot  relevant  is  designated  by  "a"  and  so  forth.     The  cross  ratio 
for  this  table  is  defined  as  the  ratio  of  two  products.  CR=AB/CD. 
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EXAMPLE  ANALYSIS  OP  CROSS  RATIO 

Independent  Variable 

Low  High 

Not  Rel_                     A  C 

Rel  or  Part  Rel         D  B 

Vftien  it  is  written  in  this  form  iti  tneahirig  is  rather 
obscure.     But^  it  is  easy  to  see  that  if  A  and  B  are  large  while 
C  and  D  are  small  the  cross  ratio  will  be  large^     The  meaning 
becomes  clearer  if  we  consider  the  odds  that  a  high  value  of  the 
independent  variable  leads  to  relevant  dbcumehts.     For  high 
values  of  the  independent  variable  the  odds  that  a  retrieved  item 
will  be  relevant  or  partially  relevant  are  given  by  B/e.     For  low 
values  these  odds  are  given  by  D/Ai     The  ratio  of  these  two  odds 
ratios  reflects  the  increase  in  odds  due  to  moving  from  a  low 
value  of  the  variable  to  a  high  value  of  the  yari^  This 
ratio  jB/e)/(D/A)  is  precisely  equal  to  the  cross  ratio.  For 
this  reason  the  cross  ratio  is  also  referred  to  as  the  odds 
ratio* 

Since  the  cross  ratio  is  always  positive  and  may  become 
infinite,  it  is  replaced  by  its  logarithm  which  has  a  more 
symmetrical  distribution  and  which,  for  samples  as  large  as  the 
ones  we  are  using  is  essentially  normally  distributed.    Thus,  in 
our  discussion  of  the  impact  of  independent  variables  in  the  very 
large  files  we  will  consistently  use  the  log  odds  ratio  as  a 
statistical  indicator.     Since  the  log  odds  ratio  is  distributed 
essentially  normally,  the  t  statistic  (that  is  the  measured  value 
of  the  log  odds  ratio  divided  by  its  standard  deyiatjLdh)  is  a 
measure  of  the  statistical  significance  of  the  observed  effect. 
At  the  same  time,  the  value  of  the  odds  ratio  itself. gives  us  a 
single  way  of  describing  the  importance  of  a  particular  variable. 

Exampie  calculation.  Each  variable  has  been  repiaced_by  an 
indicator  variable  (see  Table  16-1  of  cut  points).     The  cut  point 
for  SVAis  the  mean  7.36*     There  are  2166  cases  with  SVA  below 
this  value.  Note  that  SVA,  the  niiihber  of  relevant  items 
retrieved,  is  a  property. of  the  search  as  a  whble^  and  is_ 
inherited  by  each  of  the  items  retrieved  in  that  search.  Thus  we 
expect  that  the  items  retrieved  in  searches  with  high 
values  of  SVA  have  a  better  chance  to  be  relevant^  although  each 
particular  item  may  be  either  relevant  or  not  relevant.  In  fact, 
in  searches  with  SVA  below  the  cutoff^  1361  items  are  not 
relevant,  while  only  776  are  relevant^ 
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The  contingency  table  looks  like  this: 


SVA 


BELOW  M 


ABOVE  M 


NREL_ 
BEL  PREL 


1361 
805 

2166 


776 
1899 

2675 


Odds  Above  mean 
BeloW  mean 


Ratio 


776 

(4.137)  =  1.42 


805 


2137 
2704 

4841 


1899/776 
805/1361 

1899  X  1361  =  4.137 


STD  ERROR   (from  BMDP)  =  0.062 
t  value  =  24.395 

Hote:     0.062  is  the  std  error  assuming  a  value  of  in  (cross 

ratio  )^0.  The  t  value  is  calculated  on  the  assumption 
that  it  is  0 . 


This  is  an  example  of  a  statistically  significant  (t  larger  than 
2)  result  with  no  meaning.     It  says  that  items  retrieved  in  cases 
with  a  high  number  of  relevant  retrieval  items  are  much  more 
(factor  4.137)  likely  to  be  relevant. 


The  log  odds  ratio  has  been  used  because  it  is  resistant  to 
two  types  of  Sample  selection  bias ,  which  may  be  present  in  this 
study.    One  type  of  selection  bias  is  in  the  distribution  of 
relevance.    Although  the  average  relevance  found  in  this  project 
(about  50%)  is  similar  to  that  found  in  other  studieis>  the  end 
users  were  self- selected y  and  this  may  introduce  some  unknown 
bias  in  judgements  of  relevance ^     Similar ly>  the  searchers  were 
self  selected  and»  particularly  with  regard  to  cognitive 
characteristics/  may  hot  be  typical  of  searchers  in  general^  The 
virtue  of  the  log  odds  ration  or  of  the  cross  ratio,  is  that  as  long  aj 
the  selection  biases  of  two  variable  are  independent  of  each 
other^  the  log  odds  ratio  is  unaffected  by  the  bias.  This 
feature  makes  the  log  odds  ratio  important  in  so  called 
retrbsp€setive  clinical  studies,  where  it  is  hot  possible  to  form 
a  random  sample^     It  is  appropriate,  for  the  same  reasons,  in 
this  study,.    A  detailed  discussion  of  the  meaning  of  the  log  odds 
ratio  is  given  by  Fleiss  tl3-2]. 
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ERIC 


128 


table  13-2 


etitPdlNtS  FOR  THE  STUDY  OF  ERQSS-RAtlQ  ANALYSIS 

Each  variable  was  analyzed  tp  define  a  high  and  a  low  value^  by 
calcalating  the  mean,   using  a  satiable  data  -file.   For  e^ampple^ 
the  mean  of  search  variables  was  calculated  asing  the  -file  with 
360  searches  in  it.   The  same  file  was  t-ised  to  calculate  the  mean 
o-f  iauestian  variablesj,   since  the  mean  i  s  not  changed  by  having  9 
identical   copies  o-f   each  question  variable  in  the  -File. 


VARIABLE  TO [AL 


NO.  NAME 

FREQUENCY 

CUTPOINT 

1  SVA 

360 

7.636 

•2  SVB 

360 

7.  050 

3  SVC 

360 

10.  192, 

4  SVD 

360 

24. S7S 

5  SVE 

360 

24.31 1 

6  SVF 

360 

49. 189 

7  SVG 

360 

0.219 

S  SVR 

360 

0.572 

9  SVI 

360 

14.511 

10  SVJ 

360 

3.397 

11  SVR 

360 

10.331 

12  SVt 

360 

0.238 

13  SVH 

360 

0.218 

14  SVN 

360 

0.455 

15  QVA 

360 

33  i 575 

16  QVB 

360 

36 . 200 

17  Qve 

360 

65.500 

IS  QVD 

360 

135. 275 

19  QVE 

360 

159.825 

20  QVF 

360 

295. lOO 

21  UVA 

360 

2.492 

22  UVB 

360 

75.250 

23  UVC 

360 

4 . 000 

24  UVD 

360 

3.  150 

25  JVE 

360 

3.  625 
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User  characteristics  were  calculated  oh  a  file  cbntaihihf  the  40 
questions.   For  the  PREBRD  variable  there  were  two  missihf  cases. 
Datra  are  missing  -for  scsme  of  the  other  variables  as  well.  For 
someji   such  as  SUGGBEStED  DATABASES j    a  cut  point  has  no  meaningj 
and  is  hot  reporte-l  here.   The  APPLN  is  a  categorical   variable,  so 
the  notion  of   "above  or  below  the  cut  point  has  ho  validity." 


VARIABLE 
NOi  NAME 

1  PREBRD 

2  AF'PtN 

3  t_ANS 

4  YEARGEN 


TOTAL 
FREdUENeV 

38 

40 

4© 

4b 


eUTPOINT 
3 .  750 
7 .  050 
1 1 . 375 
14. 175 


Frequency  of 
indi  catedi 


DIALOG  use  was  cut  between   "2"  and 


as 


VARIABLE 
NO.  NAME 

1  FREQDIAL 


TOTAL 
FREQUENCY 

52 


•OUTPOINT 
2.923 


Content  variables  defined  by  the  user  were  cut  as  follows:  For 
example,   a  high  value  of  PROBDEFl   is  a  value  above  3.670  on  the 
five  point  scale. 

VARIABLE  TOTAL 

NC3.    NAME              FREQUENCY  OUTPOINT 

1  PROiPEFl            40  ■  3.670 

2  INTENTl              40  2.925 

3  PROBKNOl            40  3.625 

4  INTKNOl              40  3.475 


The  values  assigned  by  the  project  searchers,   or  by  all 
searchers^   had  different  cutpoints,    corresponding  to  the  observed 
distribution  of  those  values 


VARIABLE  TOTAL 

NO.    NAME  FREQUENCY  OUTPOINT 


1  PRQBDEF2  160  1.444 

2  INTENT2  160  1.256 

3  PR0BKN02  360  2.959 

4  INTKN02  360  1.S61 
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The  question  cal ssi f i cati oh  variables  were  treated  separately^ 
since  one  of  the  research  goals  is  to  test  for  relations  between 
them.   For  use  in  the  cross  ratio  ahalysiSji   the  cutppihts  are  as 
follows.   A  value  o?  CLTYSYN  above  mean  is  greater  than  4.575.  For 
the  second  judgeji   the  cutpoint  is  4.700 


VARIABLE 

T0TAL 

NO. 

NAME 

FREQUENCY 

CUTPOINT 

1 

GATEBORY 

40 

1.S2S 

2 

ebTYSEM 

39 

4i7lS 

3 

ebTYSYN 

40 

4.575 

4 

ebTYAVS 

40 

4.  587 

5 

SPECQURY 

40 

3.  756 

& 

SPECSUBJ 

40 

4.  691 

7 

SPECMEAN 

40 

4.387 

9 

SERCONCP 

40 

4.  400 

9 

eSNStRAN 

40 

1 .  125 

iO 

TRANSSER 

9 

0.050 

11 

FINAL 

4  b 

o.bbb 

12 

eATES2 

4  b 

1 .  55b 

13 

eLTY8EM2 

4  b 

4 . 500 

14 

eLTYSYN2 

40 

4.  700 

15 

CLTYAV(32 

40 

4.700 

la 

SPECaRY2 

40 

3.015 

17 

SPECSUB2 

40 

4.628 

18 

SPECMEN2 

40 

3.  830 

19 

SERC0NC2 

40 

2.825 

20 

C0NSTR2 

40 

0.475 

21 

TRANSER2 

40 

0.050 

22 

FINAL2 

40 

0.  000 

/ 


The  searcher  cognitive  characteristic  cutpbints  were  cal cul ated 
-from  a  -file  in  which  the  project  searchers  appeared  4  times  eachi, 
and  the  au5ciliary  Judge  was  included.    The  means  are  ^ssentailly 
ahe  same  as  the  means  -for  the  36  outside  searchers  alone.   The  LSI 
is  a  categorical   variable^   so  the  notion  o-f   "above  or  below  the 
cut  point  has  no  validity." 


VARIABLE 

TOTAL 

NO.  NAME 

FREayENOY 

OUTPOINT 

1  RAT 

52 

1 1 . 635 

2  EAS 

1 1 . 077 

3  EE 

52 

23.769 

4  RO 

52 

26.3GS 

5  AC 

52 

36. 577 

AE 

52 

33.212 

7  ACCE 

52 

12.6i5 

8  AERO 

52 

6.  865 

9  LSI 

52 

2.058 
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13*5  LOfiitlc  Refressloh 

The  key  variables  of  precision  and  recall  are  both  bounded 
by  0  and  1  {that  is  tfiey  are  percentages. )     in  this  situation  it 
is  sometimes  useful  to  perform  the  so  called  logistic 

transformation.  Each  variable  is  replaced  by  the  logarithm  of 

the  corresponding  odds  ratio.     For  exampler  the  value  4b%_is 

transformed  to  logarithm  of  40/60.  We  have  performed  multiple 

regression  analysis  of  the  transformed  values  of  precision  and 
recall  against  4  cognitive  variables r_ the  users  estimate  that 
information  will  be  ayaiiabler  the  sear^  of  using 

Dialog,  and  the  number  of  terms,  commands,  and  cycles  used  in  the 
search. 
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CHAPTER  14 •     CONSISTENCY  OF  JUDGES 


14.1  Cbhsisteney  of  Context  Variables 

All  b£  the  variables  describing  the  question  and  its  ebhtext 
were  appraised  by  at  least  two  judgeis^    The  ebhsi^stehcy  of  those 
judgements  is  a  measure  of  the  validity  of  the  ebrrespbhdihg 
concepts     Hheh  the  concept  appears  valid^  one  may  examine  the 
impact  bf  the  ebrrespbhdihg  attribute  bn  the  relevance  of 
retrieved  itemsi^    if  the  judges  agree,  and  the  intact,  as 
calculated  using  either  judges  score,  is  significant ,  we  may 
conclude  that  the  characteristic  has  some  predictive  value. 

for  variables  characterizing  the  state  of  public  or  personal 
knowledge  there  is  no  reason  to  expect  agreement,  but  the 
distribution  of  differences  is  of  interest. 

 The  results  are  summarized  in  full  detail  in  fable  14-1.  The 

judges  indicated  their  judgrae  the  table 

shows  the  differences  between  the  scor^    assigned  by  the  first 
judge  (the  user j  and  the  scores  assigned  by  the  project  sear^ 
who  served  as  the  second  judge.  "0"  indicates  perfect^^a 
For  example ,  for  the  variable  PROBDEF  there  was  exact  agreement 
in  15  cases  and  disagreement  by  two  u  in  6  cases  (4  plus  2). 

Note  that  if  each  judge  assigned  scores  at  random  the 
distribution  would  still  be  peaked  at  zero,  but  it  would  be 
approximately  triangular,  ignoring  missing  values,  a  row  for  25 
cases  would  read: 

Diff :  -4-3-2-101234 

123454321(5  exact  agreements) 


112 


133 


QdSSTiON  CONTEXT  CONSISTENCY   (N  questions=4d ) 

The  distribution  of  differences  between  judge  i( user)  and 
3udge2_{ project  searcher)  on  concepts  describing  the  context: 
For  PROBDEF  and  INTENT  the  first  judge  is  the  user.     For  the 
other  variables  two  project  team  members  were  the  judges.  A 
difference  of  0  represents  exact  agreement. 


Judge 

1  -  i 

Ttldf  e 

2 

Difference 

Variable  -5 

-4 

-3 

-2 

-1 

0 

1 

2 

3 

4 

5 

PROBDEF 

2 

7 

15 

8 

4 

3 

1 

INTENT 

i 

2 

3 

6 

12 

8 

5 

3 

PROBKNO 

2 

3 

5 

6 

5 

8 

4 

3 

3 

INTKNO 

2 

5 

6 

11 

15 

1 

PR0BDEF1  =  Problem  Befihitibh  by  User; 
INTENTl  =  Intent  by  User; 

PR0BDEF2  ^  Problem  Definition  by  Searcher; 
INTENTS  =  Intent  by  Searcher; 
PROBKNdi  =  Probiem-Ptablic  Knowledge  by  User; 
PRdBKN62  =  Problem-Public  Knowledge  by  Searcher; 
INTKN62  ^  Internal  Knowledge  of  Searcher; 
INTKNdl  ^  Internal  Knowledge  of  User 
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Discussion  of  eohsistahcy  of  Juagiients  -  Gbhtext 

We  studied  the  consistency  between  user  arid  searcher 
estimates  of  four  characteristics:  problem  definition^  intent  of 
the  user,  estimate  of  the  availabiiity  of  public  knowl^^ 
estimate  of  the  internal  knowledge*     The  first  three  of  these 
variables  refer  to  the  same  concept  while  the  forth  was  specif ic 
to  either  the  user  or  the  searcher.    The  results  are  presented  as 
histograms  of  the  difference  variables.     The  key  findings  are 
these.     For  problem  definition^  on  a  scale  of  1-5,  the  difference 
shows  a  clear  peak  at  0  with  15  of  the  40  cases  having  exact 
agreement.     14  of  the  remaining  cases  differ  by  only  1  unit. 
Thus,  we  can  say  that  in  nearly  3  out  of  4  cases  the  project 
searcher  and  the  user  agreed  within  1  unit  on  whether  the  problem 
was  weakly  or  clearly  defined. 


With  regard  to  whether  the  intent  is  narrowly  or  broadly 
defined  the  situation  is  hot  quite  so  good^     In  11  cases  there 
was  exact  agreement  and  in  a  total  of  25  there  was  agreement 
within  1  unit. 

The  situation  deteriorates  further  when  we  consider 
estimates  of  the  probability  that  information  about  the  problem 
will  found  in  the  literature  i^^  T  exact  agreement  in  only 

6  cases  and  agreement  to  within  1  unit  in  only  16  cases.  The 
most  common  occurence  (8  cases|  is  a  difference  of  2  units.  The 
user's  estimate  that  inforiaation  on  the  problem  will  be  found  in 
the  literature  is  2  units  higher  than  the  searcher's  estimate. 

With  regard  to  internal  knowledge  we^w^        expect  that  the 
user  estimates  higher  personal  knowledge  of  the  subject  than  does 
the  searcher.     This  is  borne  out  by  the  data i     in  26  of  the  cases, 
the  user's  estimate  of  his  or  her  personal  knowledge  was  2  or 
more  units  higher  than  the  searcher's  estimate. 
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fable  14-2 


eONSISTENCY  OP  QtJES^fiON  etASSIPieftTiON;    (N  questions  =  40) 

The  difference  Between  the  evaluations  given  by  two  aifferent 
judges  is  an  indication  of  the  validity  and  consistency  of  the 

question  classification  concepts  tested  in  this  pro ject»  The  

results  for  SPECQtJERy  and  SEARCb^  Bubatantial  disagrreemeat. 

The  "7"  represents  one  outlying  value*. 

Judge  1  -  Judge  2 


variable 

-5 

-4 

-3 

-2 

-1 

d 

1 

2 

3 

4 

5 

CATEGORY 

1 

1 

1 

5 

16 

9 

5 

2 

CLTYSEH 

1 

2 

5 

23 

3 

4 

2 

CLTYSYN 

1 

1 

2 

2 

29 

3 

i 

i 

SPECQURY 

3 

2 

i 

4 

9 

7 

5 

SPECSUBJ 

ii 

.7 

4 

SEARCONC 

3 

2 

_5 

11 

8 

5 

4 

1 

CONSTRAN 

1 

20 

13 

5 

1 

CATEGORSr  ^Question  Classification  (Judge. 1) :  Odmain:  Dialihdex  Categoric 
CLTYSEM  =  Question  Classif icat.^ oh  {Judge  1);  Clarity:  Semantics; 
CLTYSYN  =  Question  Classification  {Judge  IJ:  Clarity:  Syntax; 
CLTYftVS  =s  Question  Classification  (Judge  IJ:  Clarity  Score; 
SPECQURY  =  Question  Classification  (Judge  1):  Specificity;  Query; 
SPECSUBJ  »  Question  Classification  (Judge  IJ:  Specificity:  Subject; 

SPECMEftN  =  Question  Classification  (Judge  1):  Specificity  Score;  

SERCbNC;  =  Question  Classification  (Judge  1) :  Number  of  Search  Concepts; 
CONSTRAN  =  Question  Classification  (Judge  1):  Number  of  Constraints; 
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14.2  Consistency  of  Judgments:  Question  Classification 

We  can  make  a  somewhat  more  detailed  discussio 
in  Table  14-2 •This  tabl  difference  between  the 

scores  assigned  by  two  project  judges.  It  is  read  in  the  same  way 

as  Table  14-1.  On  the  estimate  of  the  number  of  dialindex  

categories  the  fudges  agreed  eractly  16  times  and  were  withi 
unit  30  times  out  of  the  40.     On  clarity  of  semantics  they  agreed 
exactly  23  times  and  were  withinl  unit  31  times.     On  clarity  of 
the  syntax  they  agreed  Exactly  29  times  and  were  within  1  unit  34 
times.     This  resulted  in  an  agreement  on  the  average  clarity 
score  which  was  perfect  in  20  cases  and  within  1  in  38  cases. 

With  regard  to  specificity  of  the  query  the  situation  is  not 
good  at  all*     Disagreement  ranges  from  -4.75  to  -F3.7.     There  is 
agreement  to  within  1  unit  in  only  17  of  the  40  cases.  With 
regard  to  specificity  of  the  subject  on  the  other  harid^  the 
situation  is  quite  good*     There  is  exact  agreement  in  19  cases 
and  agreement  to  within  1  unit  in  35  cases. 

The  mean  specificity  score  is  widely  scattered  due  to  the 
lack  of  inter- judge  agreement  oh  specificity  of  the  query. 

 The  two  judges  agreed  in  only  5  cases  on  the  hund^er  of 

search  concepts  involved  in  the  question  and  one  of  the  judges 
estimated  that  the  number  of  concepts  was  larger  in  30  of  the  40 
cases . 

Perhaps  surprisingly,  there  was  s^  agreement  (20 

cases  put  of  40)  on  the  nuBaber  of  constraints^  One  of  the  judges 

estimated  the  number  of  constraints  to  be  Higher  than  did  the 
other  in  19  of  the  remaining  20  cases ^ 

A  comparison  on  the  number  of  presuppos^^ 
in  only  9  cases  and  does  not  support  any  further  analysis. 

The  question  of  the  significance  of  inter- judge  consistency, 
in  relation  to  the  fundamental  problem  of  improving  retrieval 
performance  is  discussed  in  Section  16.4.2 
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Table  14-3^ 


QUESTION  CLASSIPICftTION:  CONSISTENCY  (N  questions  =  40;176) 

The  mean  and  standard  deviation  of  the  differences  between 
judges  are  summarized, for  the  40  project  questions  and  for  176 
questions  from  an  earlier  study.  The  fraction  of  the  cases  in 
which  the  difference  between  the  two  judges  scores  is  shown  as 
%ftgree • 


N=40 


StdPev  %Aqree 


N=176 

Meai?      StdDev  — %Agree 


eftTESQRY 

Q. 

275 

1. 

.414 

46. 

6 

6. 

.662 

6. 

.822 

64. 

2 

CliTYSEM 

e. 

3.66 

1. 

.297 

57. 

5 

-6. 

.114 

6. 

.813 

81. 

3 

ClifYSYN 

-Q. 

125 

1. 

.453 

72. 

5 

6. 

.335 

2. 

.633 

58. 

5 

SPECQORY 

e. 

742 

2. 

.225 

18. 

6 

6. 

.686 

2. 

.616 

39. 

2 

SPEC SUB J 

6. 

663 

6. 

.644 

47. 

5 

6. 

.676 

6. 

.846 

66. 

5 

SEftRCONe 

1. 

575 

1. 

.973 

12. 

5 

1. 

.426 

1. 

.917 

27. 

6 

CbNSTRftN 

6.656 

^.921 

56. 

6 

6. 

.625 

6. 

.966 

52. 

8 

CATEGORY  =  Question  Classification  C Judge  1) 
CLTYSEM  »  Question  Classification  (Judge  l)z 
CLTYSYN  =  Question  Classification  (Judge  1): 
SPECQURY  »  Question  Classification  (Judge  1) 
SPECSUBJ  s  Question  Classification  (Judge  1) 
SERCONC  =  Question  classification  (Judge  1): 
CONSTRAN  =  Question  Classif tcattcn  (Judge  1) 


:  Domain:  Dialindex  Categori 

Clarity:  Semantics ; 

Clarity:  Syntax; 
!  Specificity:  Query; 
I  Specificity:  Subject; 

Number  of  Search  Concepts; 
;  Number  of  Constraints; 
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14.3  Additional  Data  on  Question  Classification 

We  have  given  the  detailed  analysis  of  inter-judge _ 
consistency  for  the  AB  questions  in  this  project.  In  addition, 
data  wer€5  available  for  176  questions  from  an  earlier  project, 
^ich  help  to  increase  bur  confidence  in  the  validity  of  these 
question  classification  concepts^  The  data  are  summarised  in 
fable  14-3  r  ^ere  we  give  thci  mean  and  standard  deviation  of  ^  the 
difference,  and  also  the  percentage  of  cases  in  which  there  is 
exact  agreements     As  explained  earlier,  the  "hull  hypothesis" 
would  give  exact  agreement  in  26%  of  the  cases i 

Two  of  the  variables  show  agreement  in  more  than  55%  of  the 
cases,  for  both  projects:  CLTYSEM  and  CLTYSYN,  which  we  regard  as 
clearly  significant. 
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CHAPTER  15^     OVERLAP  OF  SEARCHES  AND  RESULTS 


Each  question  was  searched  by  5  outside  searchers.     It  is 
interesting  to  calculate  the  degree  to  which  they  "agreed"  in 
their  treatment  of  the  questions.    As  described  in  sections  5i 3. 3 
and  5.3-4r  an  asyxnmetrical  measure  of  overlap  may  be  defined 
between  any  two  searchers  of  the  same  question.     This  yields  4S 
{questions)  x  5  (outside  searchers)  x  4  (other  searches  t^  which 
it  may  be  cdmpared_)_=  800  pair  comparisons.    The  specific 
variablesare  STOVLP(I,J)  -  the  overlap  in  search  terms  for 

searches  I  and  J  of  a  given  question;  RPRbVLP(Ir J)  -the  

cbrrespbhdihgoverlapin  relevant  or  partially  relevant  retrieved 
itemsr  arid  RETOVLP(Ir J)  --the  corresponding  overlap  in  a 
retrieved  items.  Note  that  although  each  search  corresponds  to  ah 
individual  searcher^  the  riun^ers  I  and  J  are  assigned 
arbitrarily,  and_ simply  run  from  1  to  5  for  each  question.  They 
do  hot  play  a  role  in  the  analysis r  and  are  used  only  in  the 
explahatioh  of  how  the  overlap  is  calculated. 
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Section  15.1,  Overlap  Sttidies 

 Tfie  relations  between  se^  ^re  characterized  by  the 

overlap  in  search  terms  used  and  in  retrieved  items  among 
various  searches  of  the  same  question*    We  use  the  asymmetrical 
measure  of  overlap  given  by  the  intersection  of  two  sets  divided 
by  the  first  named  of  the  two.     Thus,  for  the  five  outside 
searches  of  any  question  there^^a^    20  values  of  the  overlap  in 
search  terms  used  and  another  20  values  of  overlap  in  items 
retrieved.     For  exan^le,  if  search  #2  used  six  search  terms, 
search  #4  used  seven  search  terms,  and  they  had  three  in  common 
then  the  search  term  overlap  measures  are:     STOVLP( 2; 4)^3/6, 
SSTdVLP(4  7  2)=3/7. 

There  are  20  values  of  the  search  term  overlap  because  each 
of  the  five  searches  may  be  compared  with  the  other  four. 

Simiiariy  if  the  retrieved  items  for  iearch  #2  number  144, 
the  retrieved  items  for  search  #4  number  72  and  they  have  36  in 
common. the  two  overlap  measures  are  RETOVLP(2;4  )=3i/4  and 
RETOVLP(47  2)=l/2. 

There  are  three  kinds  of  statistical  questions  that  we  have 
addressed  in  this  analysis: 

li     What  is  the  distribution  of  the  value  of  the  search  term 
overlap  for  all  of  the  20  x  40  =s  860  possible  cases? 

2i^    What  is  the  distribution  of  the  retrieved  item 
overlap  for  ail  of  the  20  x  40  =  800  cases? 

3i     Is  there  a  significant  correlation  between  the  overlap 
in  retrieved  items  and  the  agreement  in  search  terms  over 
these  800  cases? 

Data  are  presented  in  histograms  showing  the  number  of  pairs 
r  which  the  degree  of  overlap  falls  into  a  specified  range.  If 
reement  among  the  searchers  were  large  most  of  these  values 
Id  be  close  to  the  maximum  possible  value  of  1,  and  the  data 
Id  lie  close  to  the  lower  edge  of  the  histogram. 
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AGREEMENT  ON  SEARCH  TERMS   (N  OVERLAPS  =800) 

The  distribution  of  searcher  agreement  on  the  terms  used  to 
search  a  question  is  displayed  in  a  histogram.     I £  there  were 
exact  agreement    the  overlap  would  be  1.0.    The  observed 
distribution.     In  11^1%  of  the  cases  there  was  less  than  5% 
agreement  on  search  terms.     In  44,3%  of  the  cases  agreement  was 

20%  or  less.  The  horizontal  is  marked  in  percentage  of  all  cases 

having  the  designated  range  of  overlap  values.  Overlap  is 
between  50  and  55%  in  8.6%  of  the  cases ^ 


AT  tEAST 
BUT  NQT  OVER! 

o.osbbbbb 
b.ibbbbb 
b.isbbbb 

0.20000b 
0.2SOOOO 
0.300000 
0.350000 
0.400000 
0.4S0000 

o.sooooo 
o.ssoooo 

0. £00000 
b. £50000 

b.7bobbo 
o.75bbbb 

0.800000 

o.ssoebb 

0.900000 
0.9S0000 
1.00000 


0.00000 


FREQ 
29 
70 
B2 
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97 
52 
72 
33 
47 
S 
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24 

a 

10 
13 

3 
0 

b 

0 

12 


z 
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1.3 
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00.0 
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TOTAL      800  100.0 
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IS 
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Footnote:     X  -  approx  1.5  case 
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Table  15-2 i 


AGREEMENT  ON  RETRIEVED  ITEMS  (N  OVERLAPS  =  800) 

The  distributibh  of  searcher  agreement  as  measured  by 
retriesved  (esvaluated)  items  is  displayed  in  a  histogram^  If 

there  were  esxact  agreement^  the  overlap  would  be  1.  For  66.4%  of 

the  possible  combination  the  overlap  is  not  more  than  10%.  The 
average  overlap  is  16*6%  and  the  distribution  is  strongly  skewed. 


AT  LEAST 
BUT  NOT  OVER! 

o.osboobo 

CiOOOOO 

o.isooeo 

0.209000 
O.2S0000 
0.300000 
0.3SOOOO 
0.400000 
0.450000 
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4.4 
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3.4 
1.3 
2.4 
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1.1 
0.9 
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0.8 
1.9 
0.5 
0.6 
1.0 
0.4 
0.4 
5.1 


X  =  approximately  12  cases. 
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Table  15-3^ 


OVERLAP  OP  RELEVANT/PARTIALLY  RELEVANT  ITEMS         OVERLAPS  =800) 


 The  distribution  of  searcher  agreement  as  measured  by 

relevant  or  partially  relevant  retrieved  items  is  displayed  in  a 

histogram.  In  63.9%  of  the  cas^s  the  agreement  is  not  more  than 

10%^  The  mean  value  of  the  overlap  is  18%  and  the  distribution 

is  strongly  skewed. 
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0.0500000 

b.ibbooo 

0.150000 
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Section  15 .2  Hben  searchers  agree 

 We  gave  particular  attention  (Table  15-4.1)  to  the  3i689 

items  that  were  retrieved  by  the  outside  searchers.     Of  these, 
924  (25%)  were  relevant,  972  (26%)  were  partially  relevant,  and 
1793  (49%)  were  hot  relevant.     We  then  broke  the  items  down  into 
those  which  were  retrieved  by  only  a  single  searcher^  these 
retrieved  by  2  searchers  and  so  oh.      The  effect  of  multiple 
recovery  is  expressed  in  odds  ratios  in  Table  15-4.1)  In  the 
group  as  a  whole  the  odds  for  relevant  as  against  not  relevant 
are  approximately  5  to  10  (924/1793).     In  items  that  were_ 
retrieved  only  once  the  corresponding  odds  are  4  to  10.     In  items 
that  are  retrieved  twice  they  are  lb  to  10.     For  items  that  have 
been  retrieved  3  or  mora  times  the  odds  ar^ 

relevance  as  opposed  to  not  relevance.  The  more  often  an  item  is 
retrieved,  the  more  likely  it  is  to  be  relevant. 

 This  is  an  important  finding i  it  s^^ 

searchers  disagree  substantially  in  the  items  that  they  retrieve, 
when  they  do  agree  they  are  likely  to  be  producing  relevant 
itemsi     This  suggests  that  one  possible  super-strategy  for  the 
conduct  of  on-line  searches  is  to  have  several  independent 
searchers  work  on  the  problem  and  to  examine  first  the 
intersect ion  of  their  retrieved  sets. _  If  a  single  searcher  has 
worked  on  the  problem  the  odds  for  relevance  vs.  not  relevance 
are  less  than  1  to  2,  if  3  or  more  agree  on  a  retrieved  item 
then  the  odds  switch  to  be  almost  the  reverse  (that  is,  3 

to  2  in  favor  of  relevance).     It  would  be  interesting  to  

speculate  on  the  economics  of  going  to  such  a  multiple  search 
strategy  in  real  world  situations. 


124 

145 

o 

ERIC 


1 


RELEVANCB  OF  DUPLICATED  ITEMS   (N     unique  items;   Out"sftfe  =  3689) 

Another  measure  of  overlap  is  based  on  the  fact  that  the 
overlap  in  retrieved  items  is  due  to  items  that  were  retrieved 

more  than  once.     This  table  is  a  cross-tabulation  of  the  

relevance  of  an  item  by  the  number  of  different  outside  searches 
in  which  it  was  retrieved.     For  eacampie^  of  1602  items  retrieved 
in  only  one  outside  search  405  were  judged  relevant;  of  128 
retrieved  in  exactly  3  outside  searches,  54  were  judged 
relevant. 


EVAL 


Number  of  times  retrieved 
2  3  4 


TOTAL 


RIL 
PREL 
NREL 
NEVAL 


620 
750 
1544 


185 
129 
174 


91 
59 
56 

d 


24 
30 
19 
d 


4 
3 

d 
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924 
972 
1793 


2914 


488 


206 


3689 
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ODDS  OF  RELEVftNCS  VERSUS  NUMBER  OF  TIMES  RETRfEVED 

in  tfiis  tab3.c^  we  focus  attehtibh  on  the  rislative  odds  that 
an  item  is  relevant  versus  non-relevant^     To  sharpen  the  analysis  (in 

this  section  of  the  study  only)  partially  relevant  items  are  

disregarded^  Thus  the  total  of  all  retrieved  relevant  items  (924) 
is  divided  by  the  total  number  of  non-relevant  unique  items 
retrieved  to  give  a  base  line  it^asure  ("any  number  of 
retrievals") 


Retrieved  by 

n  Searchers 

Rei 

Not  Rel 

Odds 

Any 

924 

1793 

9.52 

1 

626 

1544 

B,4Q 
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185 

174 

1.Q6 

3,4,5 

119 

75 

1.59 
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Section  3^5.3  Goes  search  term  agreeront  explain  overlap? 

We  hav€5  prepared  a  scatterplot  agreement  as  measured 

5y  search  terms  and  as  measured  by  over tap.  It  shows  no  clear 
relation,  A  regression  analysis  shows  that  only  2 i 5% 
variation  in  overlap  of  retrieved  items  can  be  attributed  to 
overlap  in  the  search  terms. 

In  the  scatterplot  each  "I"  represents  a  particular 
cbmparisbh  of  two  searches  of  the  same  question.  When  points 
coincide  hiimbers  are  used^  and  then  letters  "A"=10,  "B"=ll  and  so 
forth.  The  bulk  of  the  data  is  concentrated  in  the  very  lowest 
row  of  the  plot,  because  overlap  of  retrieved  item  sets  is  so 
small. 

The  scales  run  from  0.  to  1.  on  both  axes.  For  exampler  "9" 
on  the  horizontal  axis  represents  6.9,  and  so  on. 
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Table  15-5 


SCATTER  PLOT  OP  SEAReH  TERM  AND  RETRIEVED  ITEM  OVERLAP 

Each  point  represents  the  overlap  in  search  terms  and  the 
overlap  inretf ieved  items  for  a  pair  of  searches  of  the  same 
question.  The  search  for  a  regression  relation  between  the 
variables  was  unsuccessful.     (R-sguared  of  2.5%} 
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CHAPTER  16.     DBTBRHINANTS  OF  RELEVANCE 


16.1    User  and  context  variables 

 One  goal  of  searchingf  in  any  large  database^  is  to  in^rove 

access  to  information  by  presenting  the  aser  a  reduced 
(retrieved)  set  of  items  of  which  as  many  as  possible  are 
relevant i    The  fraction  that  are  relevant  or  partially  relevant 
is  called  the  precision.    A  second  gfoal  is  to  retrieve  "as  many 
as  possible"  of  the  relevant  items* 

In  this  study. the  relative  recall  has  been  measured  by 
comparing  the  results  of  any  one  search  with  the  union  of  all  9 
searches  for  the  same  question. 

The  determinants  of  success  may  be  considered  on  a  search-^ 
wise  basiSf  or  item  by  item.    On  a  search-wise  basis  the  observed 
precision  and  recall  are  treated  as  dependent  variables,  and 
other  variables  are  examined  to  explore  their  influence  oh 
precision  and  recall. 

For  variables  describing  the  uses /  the  context^  or  the 
guestibh^  we  have  4Q  cases.    In  these  cases. we  may  study  the 
precision  of  the  retrieved  union  set.    Recall  is  hot  defined. 
For  cognitive  variables  we  cannot  do  a  search-wise  analysis 
because  each  searcher  did  5  or  6  searches,  which  will  differ  in 
their  precision  and  recall.    For  variables  describing  the  search 
we  have  360  cases,  with  each  characterized  by  both  a  recall  value 
and  a  precision  value. 

A  more  powerful  analysis  is  the  item-^wise  approach.    €iveh  a 
retrieved  item  that  is  relevant  or  partially  relevant  we  can 
examine  whether  the  characteristics  of  the  searcher,  the  search 
or  the  question^  user  and  context  are  different  from  those  for 
items  that  are  not  relevant. 

In  the  following  detailed  results  we  see  that  the  item-wise 
approach  yields  relations  which  seem  to  be  obscured  at  the 
search-wise  level  of  analysis.    Relations  at  the  search-wise 
level  are  all  quite  small. 


The  user  properties  are  determined  from  the  guestioii  form 
(variables  28-36)  see  Chapter         Of  these  only  28-31  can  be 
sensibly  analyzed.^    Since  only  3  users  requested  a  precise 
search,  the  data  oh  Precise  versus  Broad  are  of  doubtful 
iz^brtahce.     The  Application  variable  has  6  distinct  values  (e.g. 
undergraduate  1^  faculty,  etc.).    The  language  restriction  has  only 
two  values.     Similarly,  the  time  restriction  variable  hacl  only 
two  values. 

The  intact  of  these  variables  is  studied  in  two  ways.  Where 
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the  variable  is  essentially  binary  we  use  the  cross  ratio  as 
described  in  Chapter  13.  Where  the  variable  can  take  more  than 
two  values r  we  must  use  the  analysis  of  variance r  or  s 
tabulation.  We  have  selected  the  analysis  of  variance  as  more 
appropriate,  using  the  observed  values  of  the  (search-wise) 
precision  and  recall  as  the  dependent  variables. 

For  cross  ratios^  tables  have  been  prepared  showing  the 
value,  the  logarithm^  the  standard  errorahd  the _t-value^  ft  t- 
value  greater  than  2  is  significant  at  95%  confidence.  The  reader 
should  review  the  discussion  of  Chapter  13_ch  the  meaning  of  the 
cross  ratio,  and  the  table  of  variable  in  Chapter  1,  to  recall 
the  meaning  of  values  above  and  below  the  cut  points. 
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INFLUENCE  OF  USER  VARIABLES  ON  RELEVANCE   (N  unique  it^ms^5411) 

 Odds  ratio  is  a  measure  of  1 i ke li hood •  For 

example r  in  this  study  an  item  retrieved  for  a  user  who  gave  no 

limit  on  years  is  27.9%  more  iikeiy  to  be  relevant  than  one  

retrieved  for  a  user  who  limited  the  search  to  the  last  5  years. 

Variable         Odds  ratio  Log  dddsi      Std  Error  t-value 

LANG__  0.629  -0.464  0.057  -8.234 

YEARGEN  1.279  0.246  0.061  4.041 

LANG  »  Limit  Retrieval  by  Language  of  Publication; 
YEARGEN  =  Limit  Retrieval  by  Year  of  Publication; 


ERIC 


We  see  that  the  language  variable  show  ah  Iteznwise  influence  -  the 
chance. that  a  retrieved  item  will  be  judged  relevant  or 
partially  relevant_is  reduced  by  37%  if  the  range  of  languages  is 
hot  restricted  to  English,  The  chance  that  a  retrieved  item  will 
be  judged  relevant  or  partially  relevant  is  enhanced  by  28%  if 
the  there  is  no  restriction  given  on  the  year  of  publication. 
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Table  16-2 


INFLUENCE  OF  USER  VARIABLES  ON  REiiEVANCE  (N  questions  ^  46) 

The  question-wise  analysis  of  the  impact  of  user  variables 
oil  relevance  shows  ho  significant  difference  in  the  mean 
precision  for  the  various  groups  of  questions .Data  are 
represented  in  parallel  histograins  for  each  of  the  three  user 
variables  that  can  be  treated  in  this  way.  The  data  are  analyssed 
by  analysis  of  variance  to  determine  whether  the  differences 
between  groups  are  significant  compared  to  the  differences  within 
groups;  The  dependent  variable  in  every  case  is  the  precision  of 
the  union  of  retrieved  sets  of  items,  in  the  summary  tables^  the 
following  statistics  are  given  separately  for  each  group:  raean^ 
Standard  deviation,  ah  alternative  estimate  of  the  standard 
deviation,  the  standard  error  of  the  mean,  the  upper  and  lower 
limits  of  the  dependent  variable,  and  the  number  of  cases. 
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Table  16-.2    (continued  p. 2) 


HiSTdSRAH  OF  t  aVH        t  .         .  CASES  DIVIDED  INTO  6R0UPS  BASID  ON  WM  -  !5F  t  LANS  t 

ENGLISH  OTHER 

niDPOlNTS 
1.020} 
0.9S0>t 
0.900) 
0.840) ttt 
3.780) 


0.720) ttt  ttt 
0.660)tm  t 
0.&00)t 

o.»o)n 

0.4B0)t  t 

0.320)tttt  ntt 

0.3S0)M  t 

0.300)tt  t 

0.240)t  it 

0.180)  t 

0.120)tt  tt 
iROdP  HEANS  ARE  DENOTED  BY  H'S  IF  THEY  COINCIDE  KITH  t»S,  N'S  OTHERHISE 

nm  '0^562  0.«4 

STD.DEV.  0.247  0.227 

R.E.S.D.  0.269  p.23i 

S.  E.  N.  0.049  0.pS9 

HAxinun      1.000  0.735 

HIHIHUfi  0.115  6.107 
SfiHPLE  SIZE      25  15 


HISTOGRAH  OF  t  QVH 


CASES  DIVIDED  INTO  6R0UPS  BASED  ON  VALUES  OF  t  YEAREEN  t 


LAST  SYR 

niDPOINTS 
1.020) 
0.960) 
0.900) 
0.840) 
O.780) 
0.720)t 


NO  LIMIT 


tt 


15 


ttt 
ttt 


0.660)t 
0.600) 
O.S40)N 

0.480)  ii 

0.420) tt  m 

0.360)  III 

0.300)  III 

0.240)  I 

9.mi  t 

O.120)  tttt 
GROUP  HEANS  ARE  DENOTED  BY  H'S  IF  THEY  COINCIDE  KITH 


it 

t 

t 

H 
t_ 
tt 


tt 


HEAN 
STD.DEV. 
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All  oE  the  effects  of  user  variables  disappear  at  the  search 
wise  levels  where  analysis  of  variance  shows  no  significant 
difference  in  the  overall  precision  of  the  retrieved  set  with 
variation  in  the  purpose  of  the  search^  the  language  desired,  on 
the  restriction  on  yearly  The  mean  precision  of  the  union  is  48.5% 
for  graduate  students.  The  mean  precision  of  the  union  is  49.7%  for 
faculty.     The  mean  precision  of  the  union  is  65.8%  for  Other,  but 
the  sample  is  only  three  cases. 

 The  situation  is  essentially  the  same  when  w^  the 

context  variables.  The  only  significant  and  important  results  are 
in  the  cross-ratio  analysis.  In  particular,  sinca  there  are  two 
judges  for  each  characteristics,  we  could  not  find  a  way  to 
combine  the  judges'  scores  to  do  ah  analysis  of  variance. 
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Table  16^3 


iNPLQENGE  OF  eONTEXf  VARIABLES  ON  REEEVANCE  (N  distinct  items  found 
by  outside  searchers  »3689) 

There  are  four  context  variables  and  two  guasi-cbntext 
variables  (estimates  of  context  by  the  project  searchers}.  The 
influence  of  these  on  the  relevance  of  retrieval  items  is 
measured  by. the  cross  ratio.     Values  of  t  greater  than  2  are 
statistically  significant.     Note  that  problem  definition  has  a 
positive  impact  whether  estimated  by  users  or  project  searchers. 
Estimations  of  intent  are  inconsistent  in  their  impact  on 
relevance.     The  strongest  positive  effect  is  due  to  the  estimated 
availability  of  relevant  items. 


eut 

Variable 

Ratio 

Log 

Std  Error 

t 

N 

3.67 

PROBDEFl 

1.172 

0.159 

0.067 

2.375 

3689 

2.93 

INTENTl 

0.742 

-0.299 

0.066 

-4.512 

3689 

3.63 
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2.102 
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0.068 

11.139 

3689 

3.47 
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1.089 

0.085 

0.066 

1.283 

3689 

3.25 

PR0BDEF2 

1.023 

0.022 

0.108 

0.208 

2165 

2.83 

INTENT2 

1.328 

0.284 

0.106 

2.655 

2165 

2.96 

PRdBKNb2 

1.112 

0.106 

0.077 

1.376 

3689 

1.86 

INTKN02 

0.909 

-0.095 

0.067 

-1.426 

3689 

PROBDEFl  »  Problem  Definition  by  User; 
INTENTl. =  Intent  byUser; 

PROBKNOl  ^Problem- Public  Knowledge  by  User; 
INTKN01_=  Internal  Knowledge  of  User; 
PR0BpEF2  »_ Problem  Definition  by  Searcher; 

INTENT?  «  Intent  by  Searcher;  

PR0BKN02  «  Problem- Public  Knowledge  by  Searcher ; 
INTKNb2  s  Internal  Knowledge  of  Searcher; 

Note:  The  number  of  cases  is  different  for  PR0BpEF2  and  INTENT 2 
because  these  have  been  coded  as  searcher  characteristics 
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The  variable  '"ihteht"  as  judged  by  the  user  has  a  negative 
Impact  on  the  chance  that  a  retrieved  Item  will  be  judged 
relevant.  When  the  Intent  variable  Is  below  2.93  there  Is  a  26% 
decrease  In  the  chance  that  a  retrieved  Item  will  be  judged 
relevant 


An  item  retrieved  in  response  to  a  question  for  which  the 
user  judges  that  there  is  substantial  public  knowledge  (above 
3.63)  is  116%  more  likely  to  be  judged  relevant  than  one 
retrieved  in  response  to  a  question  on  which  the  public  knowledge 
is  judged  to  lie  below  this  cut  point. 

Intent  as  judged  by  the  project  searcher  shows  an  opposite 
correlation  with  the  chance  of  relevance. 


Problem  knowledge^  as  judged  by  the  searchers  in  general  

shows  a  positive  influence^  but  at  a  lower  level  of  si^ 
and  impact.  An  item  retrieved  in  response  to  a  question  for  which 
the  searcher  judges  that  there  is  substantial  public  knowledge 
^above  2.96)  is  11%  more  iikeiyto  be  judged  relevant  than  one 
retrieved  in  response  to  a  question  on  which  the  public  knowledge 
is  judged  to  lie  below  this  cut  point.  Note  that,  because  users 
were  more  confident  that  public  knowledge  would  be  found,  the  cut 
point  for  the  users  was  higher. 
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16.2  Impact  of  Question  Characteristics 


Variables  66  -  80  refer  to  presumed  characteristics  of  the 
question.     We  can  consider  two  questions:     do  the  two  judges 
agree  on  the  values  of  these  characteristics;  do  the  values 
assigned  by  the  judges  have  similar  impact  on  the  relevance  of 
retrieved  items •  There  is  ho  way  to  assign  a  single  numerical 
measure  of  the  impact r  since  there  are  two  judges  who  do  hot 
always  agree.  We  can,  however ,  search  for  significance  by 
preparing  a  table  showing  the  t-value  of  the  cross  ratio  test, 
for  the  determination  of  evaluation  by  the  score  as  judged  by 
each  of  the  two  judges.  Where  these  values  are  consistent  and  of 
the  same  sign,  we  may  conclude  that  the  variable  has  some 
consistent  effect  on  the  chance  that  a  retrieved  item  is 
relevant. 
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Table  16-4 

eROSS  RATIO  TEST:  IMPACT  OP  QUESTION  CHARftCTERliSTICS   (N  items=3689) 

Significance  and  sign  of  the  impact  of  question  characteristics 
oh  the  relevance  of  retrieved  items,  ft  positive  value  larger  than 
2  indicates  a  significant  positive  intact  of  the  corresponding 
variable  on  the  cross  ratio;  A  characteristic  is  regarded  as 
significant  only  if  it  is  significant  for  both  judges,  and  with 
the  same  sign. 


Judge  1 

Judge  2 

CATEGORY 

1.579 

-6.545 

CLTYSEM 

0.344 

-4.410 

CLTYSYM 

4.345 

0.699  * 

SPECQURY 

-11.770 

1.306 

SPECSUBJ 

-4.308 

-1.366 

smem 

1.045 

-7.457 

CONSTRftN 

1.606 

3.586  * 

TRANSSER 

0.726 

2.032  * 

CATEGORY  ^  Qtiestibn  Classif xcatxbn  (Judge.  1 
CLTYSEH  =  Question  Classification  (Judge  1) 
CLTYSYN  »  Question  Classification  (Judge  1) 
SPECQORY  ^  Question  Classification  (Judge  1 
SPECSOBJ  s  Question  Classification  (Judge  1 
SERCONC  s  Question  Classification  (Judge  1) 
CONSTRAN  =  Question  Classification  (Judge  1 
TRANSSER  =  Question  Classification  (Judge  1 
CftTE(32  =  Question  Classification  (Judge  2) : 
CLfYSEM2  =  Question  Classification  (Judge  2 
CLTYSYN2  ^  Question  Classification  (Judge  2 
SPECQRY2  »  Question  Classification  (Judge  2 
SPECSUB2  =  Question  Classification  (Judge  2 
SERC0NC2  =_Questibn_Classif icatibn  (Judge  2 
C0NSTR2_s  Question  Classification  (Judge  21 
TRANSER2  ^  Question  Classification  (Judge  2 


):  Domain:  Dialindex  Categories ; 
:  Clarity :  Semantics ; 
;  Clar  i t y :  Syntax ; 
):  Specificity:  Query; 
):  Specif icity:  Subject; 
:  Number  of  Search  Concepts; 
) :  Number  of  Constraints ; 
) :  Presuppositions :  Number  Transfer 
Domain:  Dialindex  Categories; 
):  Clarity:  Semantics; 
) :  Clarity:  Syntax; 
) :  Specificity:  Qtaery; 
):  Specificity:  Subject; 
):  Number  of  Search  Concepts; 
:  I^umber  bf  Constraints^ 

Pres*tppo^;itibns:  Number  Transfez 
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None  of  the  question  classification  variables  met  the 
simultaneous  test  that  they. have  significant  impact  oh  the  chance 
that  a  retrieved  item  is  relevant ,  and  that  the  impact  have  the 
same  sigh  for  both  judges.  The  variables  showing  the  same  sign  for 
both  judges  are: 

 An  item  retrieved  in  response  to  a  question  for  which  the 

clarity  of  syntax    is  high  is  more  likely  to  be 

judged  relevant  than  one  retrieved  in  response  to  a  question  on 

v^ich  it  is  low. 

fth  item  retrieved  in  response  to  a  question  with  a  high 
number  of  constraints  is  more  likely  to  be  relevant  than  one 
retrieved  in  response  to  a  question  oh  which  this  variable  is 
low. 

An  item  retrieved  in  response  to  a  question  for  which  the 
number  of  concepts  is  high  is  more  likely  to  be  judged 
relevant  than  one  retrieved  in  response  to  a  question  on  which 
this  variable  is  low. 
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16.3  Impact  of  Searcher  Characteristics 

The  file  of  3689  items  retrieved  by  outside  searchers 
provided  the  most  powerful  tool  for  investigating  the  influence 
of  searcher  characteristics  on  the  chance  that  a  retrieved  item 
will  bereieyant  or  partially  The  test  is  the  cross- 

ratio  test  r_  described  in  chapter  13.  All  possible  explanatory 
variables  were  examined  (in  factr_ali  88  variables  other  than  the 
evaluation  were  considered ij  in  the  following  tables  ail  those 
variables  showing  a  significant  and  meaningful  impact  are 
summarized.     We  include,  because  it  is  of  some  general  interest, 
the  frequency  of  recent  expierience  ih  using  the  DIAiiOG  system, 
which  is  found  not  to  have  a  significant  intact  on  the  chance 
that  retrieved  items  will  be  relevant. 

A  illustrative  table  of  the  cross-ratio  test  for  every  variable 
that  could  relate  to  the  relevance  of  unique  items  (whether  it 
could  be  explanatory  or  not)  is  given  in  Table  16-14  at  the  end 
of  this  chapter.) 
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Table  IS^S 


CROSS  RATIO  TEST:   IMPACT  OP  SEARCH       CHAMCtERISTICS  (N 
items r  with  duplicates  =^4841) 

/I         .  _  .  .      .  _  .  . 

The  cross  ratio  test  is  used  to  evaluate  the  impact  of  cognitive 
characteristics  oh  the  relevance  of  retrieved  items i  Bigh  RAT 
scores  increase  chance  of  relevance  by  65S*    High  EAS  serves  _  _ 
depress  it  by  11%.    AC  enhances  choices;  CE  depresses  them.  ACCE 
is  a  useful  combined  measure. 


Variable 

Ratio 

Log 

std  error 

t-value 

FREQDIAL 

1.104 

0.099 

0.067 

1.479 

RAT 

1.645 

0.498 

0.058 

8.596 

EAS 

0.887 

-0.120 

0.059 

-2.022 

CE 

0.735 

-0.308 

0.059 

-5.319 

RO 

0.910 

-0.094 

0.060 

-1.579 

Ae 

1.251 

0.224 

0.062 

3.619 

ACCE 

1.275 

0.243 

0.059 

4.119 

FREQDIAL  ^  Frequency  of  use  of  the  Dialog  system 
RAT  ^  Remote  Associates  Test  Score  (Searcher); 
EAS  =  Employee  Aptitude  Survey  Score  (Searcher ) ; 

CE  =  Leaning  Style  Inventory  (Searcher):  Concrete  Experience  Score; 

Rp  =  Leaning  Style  Inventory  (Searcher):  Reflective  Observation  Score;  

AC  =  Leaning  Style  Inventory  (Searcher )s  Abstract  Conceptualization  Score 
ACCE  =  Leaning  Style  Inventory  (Searcher):  Abstract  Conceptual izat ion/con 
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ftn  item  retrieved  by  a  searcher  haying  a  high_ score  on  the 
remote  Associates  Test  Is  64.5  percent  more  likely  to  be  relevant 
or  partially  relevant  than  one  retrieved  by  a  searcher  with  a  low 
score • 

 An  item  retrieved  by  a  sear cher  wi th  a  high  value  for  the 

composite  learning  style  score  called  AC  minus  CE  is  27.5%  more 
likely  to  relevant  or  partially  relevant  than  one  retrieved  by  a 
searcher  with  a  low  score. 

There  is  a  marginally  significant  (t»2.62}  indication  that 
ah  item  retrieved  by_ a  searcher  with  a  high  value  oh  the  Employee 
Aptitude  Survey    is  11%  less  likely  to  be  relevant  than  an  item 
retrieved  by  a  searcher  with  a  low  score. 
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16.4  Impact  of  Search  Characteristics 


 In  this  section  we  consider  two  very  different  kinds  of 

search  characteristics.  The  firsts  o  in  the 

training  and  selection  of  search  intermediaries,  is  the 
characteristics  of  the  search  process  ^commands/  terms  used^ 
cycles  etc.)  which  might  explain  the  relative  success  of  some 
searches • 

The  second  is  theanalysis  of  the  four  different  types  of 
project  search,  for  relative  effectiveness.  The  project  searches 
range  from  ah  approximation  to  the  usual  practice  (except  that 
the  interview  was  constrained  to  a  fixed  protocol,  to  ehsilre 
uniform  treatment  of  searches)  to  a  "dumb"  search  using  only  terms 
from  the  guery  (but  with  some  searcher  Input  to  define  the 
appropriate  Boolean  combination  of  terms.  ) 
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16. 4.1  Impact  of  search  efficiency  variables 

 One  o£  the  most  interesting  questions  to  ask  is  how  the 

performance  characteristics  of  a  search  j numbers  of  relevant ,  

partially  and  not  relevant  documents  retrieved r  total  numbei:  of 
documents  retrieved  and  most  importantly  and  precision) 

are  related  to  characteristics  of  the  search  itself  such  as 
number  of  commands^  number  of  cycles,  number  of  terms,  and  the 
time  used  in  the  search. 

This  has  been  studied  by  three  techniques,  for  ail  of  the  200 
searches  done  by  outside  searchers*    The  results  are  rather 
bleak.  Tables  16-6,7,8). 

We  shall  see  that  an  analysis  of  the  full  file  of  retrieved 
items  suggeats  that  searches  with  a  large  number  of  command 
cycles  are  more  likely  to  produce  relevant  i terns .    The  fact 

that  this  effect  does  not  show  through  in  an  analysis  of  200  

outside  searches  h -IS  several  possible  expiassations^  For  example, 

it  may  be  that  those  searchers  who  use  a  targe  nuiaber  of  cycles 
retrieve  more  relevant  items  and  more?  nohreiieviiit  items. 
Thus  their  improved  retrieval  of  relevant  items  is  hot 
revealed  by  their  own  precision  scoreis.     A'he  e<3cbnd  possibility 
is  that  we  are  uncovering  the  kind  of  bx^bII  erfect  that  only 
becomes  apparent  when  a  very  large  quant ih^  of  data  is 
accumulated . 
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Logistic  Regression  of  Precision  and  Recall 

The  key  variables  of  precision  and  recall  are  both  bounded 
by  b  and  1  (that  is  they  are  percentages.)    in  this  situation  it 
is  sometimes  useful  to  perform  the  so  called  logistic 
transforation •     Each  variab^    is  replaced  by  the_  logarithm  of 
the  corresponding  odds  ratio. exan^ler  the  value  40%  is 

transformed  to  logarithm  of  40/60.  We  have  performed  multiple 

regression  analysis  of  the  transformed  values  of  precision  and 
recall  against  4  cognitive  variables^  the  user's  estimate  that 
information  will  be  available,  the  searcher's  frequency  of  using 
Dialog,  and  the  number  of  terms,  commands,  and  cycles  used  in  the 
search. 

Considering  precision  first,  the  logistic  transformed  _ 
variable  is  named  LH.  The  most  important  explanatory  variable  is 
the  user's  estimate  that  information  is  available.     It  explains 
10%  of  the  variance  in  this  logistic  variable.    The  next  variable 
tpenter  is  the  remote  associates  score 

characteristic  of  the  searcher.     It  explains  aboutan  Iditional 
5%  of  the  variance^    None  of  the  other  possible  variables  past 
the  F  test  for  entering  the  regression.     Thus  together  these  two 
variables  explain  15%  of  the  observed  variation  in  the  log  of  the 
odds  ratio  cbrrespdhdihg  to  the  precision^    As  in  other  cases 
With  a  low  R- squared  value  we  must  conclude  that  the  bulk  of  the 
effect  is  not  explained  by  variables  included  in  this  study.  Of 
course,  we  cannot  exclude  the_ possibility  that  the  bulk  of  the 
variation  is  due  to  essentially  random  factors  highly  specific  to 
the  questions  and  the  searchers. 

 The  situation  for  explaining  the  ^  variable 

corresponding  to  recall  is  substantially  worse.     Only  1  variable 
enters  the  regression,  the  difference  in  cognitive  scores 
called  fte-CE.     it  explain  somewhat  less  than  5%  of  the  observed 
variation  in  this  logistic  variable. 
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Table 


LOGISTIC  MODEL  PGR  RECALL  AND  PRECISION  (N  searches  =  151) 

This  analysis  sought  alogistic  relation  between  recall 
(SVG)  or  precision  (SVH)  and  9  candidate  explanatory  ve^riableiSr 
at  the  search-wise  level  of  analysis.  Searches  for  which  either 
recall  or  precision  assumes  the  value  1  or  0  must  be  omitted 
because  the  transformed  variable  is  undefined.  9  explanatory 
variables  were  allowed  (describing  search  structure  and  cognitive 
characteristics.)  Only  the  significant  ones  are  reported. 

Variable  "     Model  R- squared  value 

LH  PR0BKNO1 ,  RAT  14.8% 

LG  ACCE  4.6% 


The  model  defines  "  =  ln(SVG/{l-SVG) )  and  seeks  a  relation  of 
the  form 

LG  =  Aj^xKAT+o  .  .+AgXSVk 
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For  LH  the  only  variables  drawn  into  the.  regression  are  PRDBKNOl  and 
RAT  (which  we  would  expect).    Together  they  explain  14.8%  of  the 
variance  in  LG. 

For  L6  the  situation  is  much  worse.  Only  ACCB  enters  the  regression^ 
and  it  explains  only  4.6%  of  the  variance. 

An  attempt  to  explain  t^     (search-wise)  levels  of  recall  and 
precision  with  a  logistic  model  shows  that  14. 8^        the  variation 

in  the  logarithm  of  the  odds  ^relevant  or  partially  relevant)  

versus  (non  relevant)  can  be  explained  by  the  users  estimate  of 
public  knowledge,  and  the  searchers  score  on  the  remote 
associates  testi  No  other  variables  contribute  significantly,  and 
85.2  percent  of  the  variation  remains  unexplained i 

Only  4.6%  of  the  variation  in  the  cdrrespohdihg  logarithmic 
ratio  for  the  recal  can  be  explained.  The  leading  explanatory 
cariable  is  AC  minus  C£.  \ 


We  conclude  that  the  effects  which  may  be  observed  (see 
Tables  16- 7a, b)  at  the  item~w±se  level  are  not  strong  enough  to 

predict  the  values  of  Precision  or  recall  at  the  searchwise  

level,  even  using  a  logistic  model;  In  other  words^  the  effects 
of  search  structure  are  small  compared  to  the  effects  of  searcher 
cognitive  characteristics^  which  are  themselves  small  at  the 
level  of  search-wise  analysis. 


Iteimrise  analyeis  of  the  intact  o£  search  variables 

 Some  effects  of  search  structure  can  be  seen  when  the 

analysis  is  conducte??  on  an  itemwisa  basis.  The  analysis  is 
conducted  separately  for  the  project  searches  and  for  t 
searches^  as  it  is  iikeiy  that  the  project  searches,  which  are 
four  searches  done  by  the  same  person,  will  have  some  hidden 
properties  (such  as  suppression  of  cycling  in  later  searches.) 
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CROiSS  RATIO  AMALYSIS: 

IMPACT  OP  SEARCH  CHARACTERISTICS   (N  items  «  4841) 

The  impact  of  search  characteristics  is  studied  by  orbss-ratio 
analysis  of  all  the  items  retrieved  by  outside ^^archers.  The 
nuxnber  of  cycles  is  seen  to  have  a  gositiye  effe^    on  the 
probability  that  a  retrieved  i^  relevant.  The  significant 

result  is  that  more  cycles  increases  chance  of  relevance  by  25%. 
More  of£-lihe  time  depresses  it  by  13%. 


Variable 

Ratio 

Log 

std  error 

t-value 

SVJ 

1.251 

0.224 

0.058 

3.849 

SVM 

^^68 

-0.142 

0.058 

-2.447 

SVJ  =  Number  of  Command  Cycles  Used  in  a  Search; 
SVM  =  Preparation  Time  Used  in  a  Search; 
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Table  i6-7b 


CROSS  RATIO  ANA£YS£S 

IMPACT  OP  SEARCH  CflARACTERISTieS     (N  items  =4il5) 

The  impact  of  search  characteristics  is  studied  by  cross-ratio 
analysis  of  all  the  searches  done  by  project  searchers.  The 
niimber  of  cycles  is  seen  to  have  a  positive  effect  on  the 
probability  that  a  retrieved  item  is  relevant .Number  of  search 
terms,  and  total  time  used,  have  a  negative  effect 


Variable 

Ratio 

Log 

Std  Error 

t- value 

SVJ 

1.23.4 

0.194 

0.070 

2.809 

SVK 

0.618 

-0.481 

0.070 

-6.819 

SVN 

0.637 

-0.451 

0.068 

-6.589 

syj  =  Number  of  Command. Cycles  Used  in  a  Search; 
SVK  =  Number  of  Search  Terms  Used  in  a  Search; 
SVN  =  Total  Time  Used  in  a  Search; 
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The  number  of  cycles  used  has  a  positive  impact  on 
relevance.  Items  retrieved  in  a  search  with  a  large  number  of 
cycles  are  25%  more  likely  to  be  relevant  or  partially  relevant 
than  those  retrieved  in  a  search  with  a  small  number  of  cycles. 

On  the  other  hand ^  items  retrieved  in  searches  which  take  a 
large  amount  of  preparation  time  are  less  likely  (by  13%)  to  be 
relevant . 

 For  the  project  searches  three  variables  are  found  to  be 

significant.  High  number  of  cycles  produces  a  21%  increase  in  the 
chance  of  relevance 

 High  number  of  search  terms  produces  a  39%  decrease  in  the 

chance  of  relevance 

High  total  time  produces  a  36%  decrease  in  the  chance  that  a 
retrieved  item  will  be  relevant. 


In  order  to  explore  whether  mult i variable  effects  were 
involved  we  also  studied  the  multiple  linear  regression  of 
variables  SVE,        G,^  H  on  the  two  pairs  SVJ,  K  and  SVL^  M.     In  ho 
case  is  the  R- squared  value  greater  than  5%.     In  other  words  none 
of  these  characteristics  of  the  search  has  more  than  5%  of  its 
observed  variation  explained  by  the  number  of  commands  or  command 
cycles  or  by  the  on-line  time  or  total  time  used  in  the  search. 


Linear  regression  of  recall  and  precision 

The  fact  that  the  effects  described  above  are  visible  only 
at  the  item-wise  level  was  checked  by  sttadying  the  linear 
regression  of  recall  and  precision  on  all  of  the  indicated  search 
characteristics.  The  results  show  that  the  effects  are,  in  all 
cases,  small. 
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Table  16^8 


LINEAR  REGRESSION  FOR  RECALL  AND  PRECISION  (N  searches  :=  200) 

Discussion:     This  was  a  search  for  linear  models  of  SVG  and  SVH  as  a 
function  of  the  pairs  (SVJrSVk)  or  (SVLrSVM) .     Th?  R-^quared  values 
are  all  very  small^  The  table  shows  the  fraction  of  variation  that 
is  explained  by  each  pair  of  explanatory  variables,  for  each 
choice  of  the  dependent  variable. 


Independent 

Variables 

Pepehdeht 

(SVJ,SVK) 

(SVLrSVM) 

Variable 

SVG 

O.li 

2^5% 

SVH 

2.5% 

0.7% 

SVG  =  Search  Recall; 
SVH  -  Search  Precision; 
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Given  the  various  indications^  at  the  it  em- wise  le^^li  that 
these  search  characteristics  have  an  intact  on  the  v^an^re  of 
relevance^  We  looked  for  regreiision  models  de^cribin^  precis?;oT:, 
and  recall  in  terms  of  theise  variables,  in  no  case  is  more  tni  t 
2.5%  of  the  precision  or  recall  explained  by  these  variables. 
None  of  the  models  are  significant  at  the  85%  confidence  level 

An  alternative  kind  of  search  characteristic  is  the 
comparison  of  the  project  searches #  which  ranged  from  a  strictly 
mechanical  search  to  one  involving  both  a  restricted  interview 
and  the  v^itten  question  statement.  The  key  descriptive 
statistics  are  summarized    Table  16-9 


Table  16-9 

COMPARISON  OP  STATISTICS  ON  PROJECT  VS  OUTSIDE  SEARCHES 


The  summary  statistics  for  outside  searches  and  project  searches 
are  assembled  together. 


Outside 

200  searches 

Project  160 

searches 



Mean 



Medn 

-   

Accuin 



Mean 

Medn 

ftccum 

SVH 

6.89 

3.0 

LOW 

8.57 

4.0 

LOW 

SVB 

6.63 

3.0 

LOW 

7.58 

4.0 

LOW 

sve 

iO.68 

5.0 

LOW 

9.57 

5.0 

LOW 

SVD 

24.21 

16.0 

LOW 

25.72 

17.0 

LOW 

SVE 

25.03 

2.Q 

LOW 

23.40 

3.0 

LOW 

SVF 

49.24 

25.0 

LOW 

49.12 

25.5 

LOW 

SVG 

0.20 

0.12 

LOW 

0.25 

0.17 

LOW 

SVH 

0.54 

0.57 

HIGH 

0.61 

0.65 

HIGH 

SVI 

15.75 

14.00 

13.00 

11.00 

SVJ 

3.73 

3.0 

2.98 

3.00 

SVK 

10.22 

8.0 

10.47 

3.0 

SVL 

0.26 

0.22 

0.20 

0.17 

SVM 

0.25 

0.25 

0.18 

0.17 

SVN 

0.51 

0.43 

0.38 

0.36 

SVA  =  Relevant  Items  Retrieved  in  a  Search; 

SVB  =  Partially  Relevant  items  Retrieved  in  a  Search; 

SVC  =  Not  Relevant  items  Retrieved  in  a  Search;  

SVD  =  Total  Number  of  Items  Retrieved  in  a  Search  and  Evaiuaied  by  the  U 
SVE  =  Items  Retrieved  in  a  Search  but  ^bt  Evaluated  by  the  User; 
SVF  =  Total  Number  of  Items  Retrieved  in  a  Search  ot  a  Question; 
SVG  =  Search  Recall; 
SVH  =  Search  Precision; 

syi  =  Number  of  Commands  Used  in  a  Searchi 
syj  =  Number  of  Command  Cycles  Useil  in  a  Search; 
SVK  =  Number  of  Search  Terms  Used  in  a  Search; 
SVE  =  Online  Connect  Time  Used  in  a  Search; 
SVM  =  Preparation  Time  Used  in  a  Search; 
SVN  =  Total  Time  Used  in  a  Search; 

"Accum"    Low  means  cases  accumulate  at  the  low  end  of  the 
scale. 
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The  project  searches  have,  oii  the  average^  higher  recall  and 
precision.    Yet  they  use,  on  the_ average,  fewer _cycles.  This 
underscores  the  fact  that  the  relation  between  "more  cycles"  arid 
enhanced  chance  of  relevance  is  too  weak  to  be  seen  at  the  level 
of  search-wise  analysis. 

  Project  searchers  use  less  connect  time  and  less  of f  line 

time^    Since  they  do  4  searches  of  the  same  guesticjn,  this  is  not 
surprising i    That  fact  may  also  account  for  the  decreased  number 
of  cycles  in  project  searches. 

The  characteristics  of  outside  searches  and  project 
searches  were  compared^  Recall^for  project  searches^  averages 
25%r  slightly  higher  than  recall  for  outside  searches  (20%). 
Precision  is  also  slightly  higher,  at  61%  versus  54%.  This 
underscores  the  absence  of  an  inverse  relation  between  precision 
and  recall. 


16.4.2  Inqpact  of  type  of  project  search 

The  possibility  of  a  distinction  among  the  four  types  of 
project  search  was  studied  by  analysis  of  variance,  applied  to 
the  search-wise  precision  and  recall.  This  was  the  only  case,  in 
this  study^  where  significant  effect  were  found  at  the  search-wise 
level.  The  following  tables  for  the  analysis  of  variance  are  read 
in  the  same  way  as  Table  16-2 
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ANALYSIS  OP  VARIANCE  FOR  RECALL  AND  PRECISION  OF  PROJECT  SEARCHES 
(N  searches  ^  160) 

This  is  a  study  of  the  distri^  tht  Recall  and  the 

Precision  values  for  the  fou^  searches. 
In  the  accompanying  combined  histograms  each  *  represents 
one  search  and  M  represents  the  median  value. 
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We  see  that  the  values  of  recall  tend  toward  the  lower  end  of  the 
scaler  and  ail  dlstrlbutiona  are_skewed»     Hence  the  P_^test  for 
analysis  of  variance  is  of  doubtful  applicability.     The  data  do 
suggest  that  the  observed  differences  in  the  means  are 
statistically  significant. 

 Search  type  1  has  the  highest  mean  recall  at  0.321 


P-test  probability  0.0506  or  0.0497  (depending  on  the  choice  of 
test ) . 


The  data  for  precision  are  seen  to  accumulate  at  the  high 
endf  presumably  due  to  smallretrieved  sets  with  100%  relevance. 
The  distributions  are  similar  to  the  eye,  and  the  analysis  of 
variance  confirms  that  impression. 

in  other  words^  analysis  of  variance  to  compare  the  recall 
levels  of  the  four  types  of  project  searches  reveals  a 
significant  differences  (at  95%  cbnf iderc?e) .  The  recall  of  method 
1  (Interview  based)  is  32%;  the  recall  of  method  3  (dumb  search 
base  don  terms)  is  18%. 

A  corresponding  analysis  for  precision  reveals  no 
significant  difference 

Intact  of  multiple  retrieval  on  relew^nce. 

We  recall  here  the  results  of  Chapter  15,  Ta^  15-4.1,4.2. 
They  show  that  the  more  often  an  item  sS  retrieved  by  different 
outside  searchers  the  more  likely  it  is  to  be  relevant. 


followed  by  type  4 
type  2 
type  3 


0,2b4 
0.231 
0.182 
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Table  16-11 


INFLUENCE  OP  MULTIPLE  RETRIEVAL  ON  RELEVANCE 

The  chance  that  an  item  will  be  relevant r  as  opposed  to  non- 
felevant,  is  strongly  affected  by  the  number  of  distinct  outside 
searchers  who  retrieved  that  object.  This  effect  has  been 
discussed  earlier,  in  relation  to  overlap,  it  is      peated  here 
because  it  has  an  impact  on  rel  that^  .  this 

analysis  the  "partially  relevant"  items  are  disregarded. 


Retrieved  by 

h  Searchers  Rel  Not  Rel  Odds 

All  924  1793  0.52 

1  62D  1544  D.4Q 

2  185  174  lo06 

3  or  more  il9  75  1.59 


159 

189 


The  odds  that  an  item  will  be  relevantr  as  opposed  to  non- 
relevant  (i*?noring  the  partially  relevant  Items)  increases 
rapidly  wlt^   the  number  of  independent  searchers  who  retrieve  it. 
The  odds  rise  from  4  to  10 r  for  an  item  retrieved only  one 
searcher r  to  16  to  10 r  for  an  item  retrieved  by  tf     e  or  more 
searchers • 
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16.5  The  cut  points  for  cross  ratio  analysis 

_        The  cutpbints  for  the  cross  ratio  analysis  are  reported  in 
Chapter  13^  Table  13-2^  They  provide  the  numerical  value  that 
distinguishes  low  arid  high  scores  on  various  explanatory 
variables.  If  they  were  to  be  recalculated  for  a  different 
selection  of  serches,  searchers  or  guestioris^  the  cut  points 
would  in  general  be  different.  However,  the  type  of  relation  that 
is  analyzed  here  would  still  appear,  possibly  with  a  changed 
numerical  value. 

The  ideal  way  to  determine  outpoints^ w         be  to  desing  a 
prospective  experiment  using  a  random  sample  of  questions  and 
searchers.  The  cutpbints  for  those  sample  would  be  typical  of 
thepbpulatibn  as  a  whole ^  and  sb,  presumably,  would  be  the 
cutpdirits  for  the  characteristics  of  the  searches  that  result.  In 
summary,  the  techriique  of  crbss-ratio  analysis  can  uhcbver 
relatibriships  which  ought  theri  to  be  tested  oh  a  random  sample 
from  the  population  of  questions  and  searchers. 
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16.6  Summary  of  statistical  tests  applied 


 Very  f aw  stones  are  unturned  in  our  search  for  expiariatbry 

relations  among  the  data.  This  is  best  seen  from  the  following 
summary  of  the  specific  programs  which  have  been  written  arid  run 
durihg^the  analysis;  The  number  of  tables  produced  by  each  run  is 
shown  in  the  last  column;  it  is  a  good  indicator  of  the  number 
of  relations  that  ha^-^  been  examined; 

In  Table  16-12  we  present  a  list  of  the  data  files  used.  In 
Table  16-13  we  list  the  analyses  that  were  done. 

Discussion  of  Table  16-^12 

 Data  were  collected  on  the  numerous  forms  shown  in  the 

appendices.     Prom  there  the  data  were  entered^  generally^  into 
dBase  files.     The  data  were  then  extracted  from  the  dBase  files 
into  ASCII  files  in  a  standard  fixed  field  format.     These  data 
refer  to  properties  of  the  context,  the  question,  the  searcher  or 
the  search.     They  are  then  joined  with  a  file  that  we  have  called 
the  backbone i     Each  case  in  the  backbone  contains  the  following 
data  elements:     question  number,  search  number,  searcher  number r 
data  base^  item  number  and  evaluation. 

Special  programs  were  written^  in  the  BASIC  language  to 
perform  the  equivalent  of  a  data  base  join  operation.     In  this 
way  the  values  of  all  the  variables  specific  to  sear^h^  searcher, 
context  or  question  can  be  added  to  the  specific  it  -.;fi  to  which 
they  refer;     In  factjp  because  of  limitations  and  space,  each  of 
these  variables  was  reduced  to  a  binary  variable,  taking  the 
value  "1"  or  "2",  according  as  the  variable  was  above  or  below 
the  cut  point  shown  in  Table  13-2. 

Thus  the  la29e  joined  files  contain  full  inf  or  v%t  ^'.»^ 
the ^ backbone  data,  together  with  a  flag  for  each  of      ^  ^her 
variable  indicating  whether  it  was  above  or  below  the  eat  point; 
This  file  was  ^  then  reduced  in  several  ways  according  to  the  need^=^ 
of  the  analysis.     One  reduction  is  the  removal  of  duplicate 
items,  so  that  each  item  is  represented  in  the  file  only  once. 
In  this  file  the  searcher  variables  bacome  irrelevant^  as  cases 
or  records  only  represent  the  first  searcher  who  happened  to 
retrieve  the  item.     Similarly  an  item  may  be  retrieved  in 

severe''  searches,  the  search  vari^ibles  become  irrelevant.     On  the 
other   .and,  characteristics  of  the  context  and  the  question,  and 
of  the  user  remain  n^aningfui; 

A  further  reduction  was  to  separate  the  file  into  those 
items  that  had  been  recovered  by  outside  searchers,  and  those 
recovered  iri  project  searches.     These  files  ar^  needed  for  the 
detailed  analysis  of  multiple  retrieval  of  an 
item  by  outside  searchers,  or  comparison  of  the  four  types  of 
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project  search. 

The  file  called  ftDDQD^dUT  is  the  combined  data  from  all  the 
files  of  the  form  described  elsewhere  in  the  report r  containing 
data  oh  individual  questions^    The  files  called  J*  are  thii  ASCII 
files  formed  from  the  dBase  files  described  above. 

The  files  called  JOIN*  are  joins  as  described  above. 


 Pinallyr  four  data  files  were  compiled  containing  the 

fundamental  input  information  on  overlap  for  the  200  outside 
searches.     Each  of  the  records  in  these  files  contains  4  overlap 
values,  so  there  are  together  200  distinct  cases. 
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^ble  16-12 
DATA  FILES 


The  analysis  was  based  upon  a  sot  of .  files  as  described  iii 
Chapter  13.  A  more  detailed  description,  with  their  working 
names,  is  given  here. 


File  Name  Records  Description 

BACKBdNE.ODT  8956  Backbone  file  to  join 

BACKBONE. TRF  8956  "    after  transform  (R,P,NjE)  to  (1,2,3,4) 

BACKBONE. RED  5411  Reduced  backbone  file—  removing  duplicate 

BACKBONE. RTR  5411  "    after  transform  (R,P,N,Ej  to  (1,2,3,4) 

BACKBONE. SCH  360  Reduced  backbone  file  --  unique  question  and  searcher 

BACKBONE. CNT  3689  Reduced  backbone  file  ~  unique  accession  no's,  add  coui 

ADDQD.OUT  360  Combined  QUES*.OUT  file 

Jf6rM5.TXT  40  mrm  5  dbase  file 

JFbRM3.TXT  52  Form  3  dbase  file 

JFORMTS.TXT  40  Form  7  &  8  -ibase  file 

JSebNTXT.TXT  360  Scontxt  dbase  file 

JFORMll.bRG  176  Form  11  dBase  file.  From  Phase  I  project 

JFdRMll.TXT  40    "  --  Reduced  to  40  searchers  we  need 

JSCORE.TXT  52  Score  dbase  file.  Cognitive  variables 

JOIN. OUT  8956  Join  BACKBONE. OGT  with  transformed  Dbase  files 

JOIN. RED  5411  Join  BACKBONE. RED  with  transformed  Dbase  files 

JOIN^SCH  360  JoiA  BACKBONE. SGH  with  Dbase  files 

JOININiSCH  160  Join  BACKBONE. SCH  with  Dbase  files  —  Project  searchers 

JOINOUT.SCH  200  Join  BACKBONE. SCH  with  Dbase  filf  ^ide  searchers 

J0IN7.0UT  5411  Join  BACKBONE. RED  with  JFORM78.T^'  vT.TXT 


OVERLAP. EVL  200  Overlap  for  Evaluated  accessicn  n-.  .3 
OVERLAP. RPR         200  "  Rel+P.Rel 

OVERLAP. NOT        200  "  Not  Evaluated 
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PROGRAM  FILES 

Plies  are  grouped  according  to  query     litc'ir  fan  internal  working 
number)^  As  a  query  (such  as  QOi)  t;at^    ■     'ior.vted,  dlgftfs  were 
added  to  its  code,  as  in  a  Dewey  decisiCA  ncfvr        Tbus  Q0i3  was 
the  third  variant  of  the  problem  origiri&ily  £v     riS^Dd  as  Query  1. 


&t  File  Kame     Date        Ii^t  File  Description  Msrber  a£ 

Tables 

1     QOll.EWb   7/29/86    JOININ.SCH     Cr^st^les  between  12 

(SVG1,SVG2,SVG3,&S;G4)  and 
( SVHl , SyH2 , SyH3 , SVH4 ? 


Q0l2.»lb   7/29/86   JOINTN.SCSi     Histograms  for  Gl2,Gl3,Gl4,G23,  12 

G24 ,G34 ,H12 ,Hl3 ,Hl4 ,923 ,H24 ,H34 


Q013.BMD   8/13/86   JOIOTN.SCa     Analysis  of  variance  by  type  2 

SVS^SVH 

2  Q021.a©   7/29/86    JOINDOT.SCH   RegressiOT  8x14 

(RKT, . . .  ^ASRO)  X  (SVA,...,SVN) 

0022. BMD    7/31/86    JOINOOT.SCH    Cross  tables  14 

LSI  X  (SVA,  ...,S7N) 

3  QOSl.BMb    7/29/86    JOINQOT.SeH    i  digression  15 

(S7I,SVJ,SWC)  X  (SVG,SVH,SVL,SVM,SVM) 

Q032.BMD   7/29/86   JOlNOUT.SeH   Recession  15 

(SVt,3^J,Sm)  X  (SVG,SVH,SVL,SW1,SVN) 

4  Q04.E©     7/29/86    JOIIHN.SCH     Regression   7x5 

(SVR,.../QVF,QVH)  X  (UVA, . . .  ^UVE) 

5  005.  B^©     7/31/86    JOilHN.SCH     C^(»Stables  7x3 

(QVA,...,QVF,QVH)  X  (PR£BRD,APPLN,LAN5) 

6  006. BMD     7/29/86    jblKDWI.SOS    l^essions  9 

(SVF,...,SVN)  X  ERBQDIAL 

7  Q07i.Bi©   7/29/86   JC^tJSUT.SCH   Regfessicms  6x6 

(SVA,SVB,SVe,SVD,SVG^SVH)  X 
(SVI  ,SVJ,  SVK,  SVL,  SVM,  SVN) 


Q072.BMD    7/29/86    JOININ.SCH  "  "6x6 
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8     QOB.EWD     7/29/86    JOINIK'.SCH     R^essibn  _     _    7x8 

INrEtiri,FR(BE^i,  INTKbloi } 


9  Q09  BMD     7/31/86    JOININ.SCH     Histograms  15 

V51-V55 ,  V52-V56 ,  V53-V57 ,  V54-V58 , 
V59-V7d,...,V69-V8b 

10  Qld.BMb     7/29/86   JOmm.i-^M     Rep^sion  5x4 

(UVA,UVB,UVC,UVD,UVE)  X 

11  Qll.BMD     7/31/86    JOINOOT.gCB    Multiple  regression  4x2 

(SVE^^SVP^SVGiLSMH)  on  (SVJ  and  SVK) 
or  (SVL  and  SVH) 


12  Q12.ikD     7/31/86    JOIlU0t.TT.SCH    St^swise  regression  1 

(LGflH)  on  (RAT,EAS,ACCE,;^RO,FROEKtK), 
FRBQDIAL,  SVI ,  SVJ,  SWJ ) 

13  Ql3.BMb     7/31/86   BftGffflbNE.eOT  Crosstab     EVT^  x  Count  1 

15  QlS.BftS     8/i8/86    JFORM78iTJeP    | EROBra©t-nBan (PRCBEa!©2 )  |  <  STO  ?  1 

jssQNKeriKer 

16  Q16.AB8     8/'21/86    QV^IAP.EW*    Histograms    ETOL  3 

OVERLAP.  RTT  MOTTEV^ 


17  Q17.BMD     8/18/86    JOIN.SCH        Regression    SVG  x  SVH  1 

18  QlS.BMb     8/21/86    JK1RM11.C25B   Histograms  U 

V59-V7d,V6d-V?l-  ...  ;V69-V8d 

19  Q19.BMD     8/21/86    BftCKBaNEiTRP  Crbsstable     EVAL  x  SEARCHER  1 

from  B?«:KBOK!^,,ajx 

20  Q2d.BM)     8/22/86    BACKBONEI.RTR  Crosstable     E?^  x  SEmZl^  1 

frcm  BACKBONE. RED 


21  Q21.B^C)     8/23/86   JCININ.SCH     Analysis  of  Variances  of  QVH  by  3 

groins  in  APFI.N(5}  ,LAt%3(2},YEARGEN(3} 

22  Q22.a4b     8/23/86    JOINOOT.SCH    ^crtptive  statistics  c£  SVC3,SWI  2 

Select  xf  RM  >  17  and  EAS  >  13 


166 

o 

ERIC 


23  Q23.BMD     8/25/86   jCHlJ7.0OT      erosatables  2 

X  ({>ROi£iF2>ipE3^2j  «n.th 
nsw  ccttpoihts  (3;25>2.825) 

24  QIN.BMD     7/31/86   JOININ.SCH     Descriptive  statistics  for  all  90 

variables,  means,  modes  etc 

25  QOOT.Bffi)    7/31/86    JOTNOtJT.SCaj  "  89 

26  TEFK03.BMb  10/10/85  JSCORE.TXT     Distribution  of  searcher's  i 

psycholo^ 

27  raSKd4.B^©  10/15/85  JSeeRE.ECT     setter  ^6t  R7a?  X  ESkS  i 

28  fEFKOS.BMD  1/15/86    JSCCRE.ECr     absstabls  8 

(BAT^EAS)  X  (CE,RO,AC,AE) 

29  ^EKOe.BM)  6/25/86    fflPDQD.OOT       Dercriptix       -  IstiCT  for  all  7 

all  Dbase      mcigfcl^  '  €U.:iD.aiS,JW 
files  JFCero  .Tg.  JS^^  HVr/8 .T}CT, JSCONTXT.TXT, 

JFORMll .T3t ,  J .  jfiE.TXr 

30  TEPk07.BM5  6/25/86  JOIN. Off  Grcssstables  EVM.  X  Xvariables  88 
TEPKb72.BMD  "  JOIN.RED  "  50 
TEFK073.BMD      "         JOIN.SCH                             "  88 

31  Te7iN.H43  7/8/86  JOTSaN.OtJT  CrMStables  EWVL  x  Xvar  -  Inside  88 
T072IN.BMD      *'  JOINtN.RHD  "  50 

32  T070OT.H©  7/8/86  JOINOOT.OOT  Crc«stables  EVMi  x  Xvar  -  Outside  88 
T0720UT.BHD    "  JOIISOOT.HED  " 

33  TSEK08.BKE)  6/26/86    JSCONIXT.TXT  Matter  ploti  2 

(FR0BKN02,INim)2)  X  Question  # 

34  lE!HKd9.BMD  6/26/86    JSeXttJIXT.TXT  I^icriptive  statistics  of  all  1 

variables  by  questions 

-^5  lEPiaO.BMD  6/26/86    JiORMn.TJCT   Grbsstables  11 


(vari^les  in  Judge  1)  x  (variables 
in  Judge  2) 


188 


eilOttiilen  oM  tmutB  ttiat  hava  &aeh  performed 


The  range  of  possible  rislatibhs  ihv»- ^:  v  ^ed  is  shown  in 
Table  16-13.     We  briefly  summarize  what  : '-^'^'^i^.^    various  research 
guieries  involved.  A  research  query  (Q#J  i^.  uot  to  be  confused 
with  a  study  question  (of  which  there  were  40) 

Q#l  The  first  set  of  gueries^  involving  a  total  of  26 

analyses  were  the  search  for  relations  among  the  project 
searches.     We  analyzed  crosstabulat ions  of  the  precision  and 

ecall^by  type^  prepared  histograms  for  the  difference  b*^tj^e^i\ 
precision  and  recall  for  all  possible  pairs  of  types ^  and 
u ?rformed  analysis  of  variance  by  type  for  the  precision  and 

•call.     The  last  was  successful  and  is  described  in  the  report. 

Q#2  iri  analysis  of  psychological  variables  we  performed 

linear  regressions ^  with  either  variable  treated  as  independent r 
on  the  ordinal  scale  psychological  variables  (RAT, . . .^ftERO) , 
against  all  of  the  search  variables  (SVA, . . . r SVNJ . _  This  was  a 
total  of  104  analyses.     We  also  prepared  crosstabulatiohs  of  the 
learning  styles  inventory  against  all  14  search  variables.  No 
significant  results  were  found. 

Qi3  We  also  performed  regressions  of  the  variables  describing 

the  search  structure^   (SVi. . . SVK)  against  the  precision  recall 
and  time  measures.     These  -Mi  analyses  produced  no  significant 
results. 

Q#4         We  performed  35  regression  analyses  on  the  relation 
between  questions  v^  (QVA. . .QVH J  and  the  user  variables 

(UVA. . .UVE) ,  which  referred  to  user  evaluationv    Only  the  obvious 
results,  that  users  were  more  satisfied  when  they  had  more 
relevant  items  was  observed. 

Q#5   21  crosstabulat ions  were  prepared  of  question  variables 

against  the  user  variables  that  restrict  the  search.  No 
significant  results  were  found. 

Q#6  degression  of  all  search  variables  against  the  frequency 

of  use  of  dialog  was  performed  (9  regressions).     No  significant 
results  were  found. 

Q#7  All  search  variables  having  todb  with  tetrieved  items 

were  analyzed  by  regression  against  all  variables  having  to  do 
with  the  structure  and  time  of  the  search  (36  analyses).  No 
important  results  were  found.     This  analysis  was  performed 
separately  for  searches  done  by  project  searchers  and  by  outside 
searchers . 
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Q#8         Regression  analysis  for  all  project  searches  was 
performed  for  question, variables  against  context  variables  as 
defined  by  the  user,     A  total  of  56  regressions  with  no 
significant  results. 

Q#9  Histograms  were  prepared  of  the  difference  betw^^  the 

judges"  scores  for  the  context  variables  and  for  the  question 
classification  variables,  a  total  of  15  histograms.     The  results 
are  summarized  elsewhere  in  the  reports 

Q#10  20  regression  analyses  were  performed  oh  the  user 

variables  against  7onte::t  variables.  No  significant  results  were 
found. 

Q#ll  8  regressions  were  performed  to  analyze  multiple 

regression  of  search  variables  SVE..,SVH  for  cheir  dependence  oh 
either  terms  and  cycles f  or  time  used.     The  results  are  described 
earlier  in  this  chapter. 

Q#i2  Stepwise  linear  regression  was  performed  for  the 

logistic  transformed  precision  recall  for  all  outside  searches  to 
study  dependence  oh  all  searcher  characteristics  and  search 
structure  characteristics  SVI. .. SVK^  as  wel)  as  the  searchers 
estimate  of  public  knowledge. 

Q#13  For  all  outside  searches  a  crbsst^bulatioh  of  the 

relevance  by  the  number  of  times  retrieve^-  was  performed.  This 
led  to  extremely  significant  results  whicK  are  discussed 
elsewhere  in  the  report. 

was  not  assigned  to  a  task 

e#l5  i'he  distribution  of  searcher  estimates  of  public 

knowledge  was  compared  with  the  user  estiiaate  of  public 

Q#ir»  Q'^rerlbp  of  searches  was  measured  through  overlap  of 

retrj^a^/efl  itei^^^  resulting  in  three  histograms  presented  elsewhere 
in  this^  rrspbrt. 

Q#17  The  precision  recall  relationship  was  tested  by 

regression  of  precision  and  recall.    The  results  are  described 
elsewhere  in  this  report. 

Q#i8  Eleven  histograms  *7ere  prepared  on  the  difference  of  the 

scores  assigned  by  two  judges  for  the  question  variables.  The 
re&uits  are  summarized  in  the  report. 
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Q#i9iQ#20         Crosstabuiations  were  prepared  for  all  items 
retrieved  and  for  unique  items  retrieved  to  give  the  searcher 
classification  (outside  or  the  four  project  types)  by  evaluation. 
The  restxlts  are  summarized  elsewhere  in  the  report. 

Q#2i         Analysis  of    ---iance  of  precision  was  performed  by 
groups  for  user  variables  restricting  the  search « 

Qi22  Descriptive  statistics  were  f  for  a  set  of 

searchers  vfhb  had  high  scores  on  both  the  RAT  and  the  EAS.  The 
results  were  hot  significantly  different  from  those  for  all 
outside  searchers. 


Q#23  Crbsstabulatlbh  was  formed  for  the  evaluation  by  the 

project  searcher's  judgement  of  problem  definition  and  intent 
with  revised  outpoints. 

Q#24^Q#25   Ninety  tables  of  descriptive  statistics  were 

prepared  for  all  project  searchs.     89  t^«^bles  of  descriptive 
statistics  were  performed  for  all  out rLd*?^  search  Distribution 
plots  were  formed  for  atl  of  t •  t  cogni^^.i^e  variables. 

Q#27         Scatter  plot  and  reg>  ^   Lon  analyses  were  attempted  for 
RAT  by  EAS  with  no  significant  result. 

Q|28      _  RAT  and  EAS  were  crbsstabulated  with  the  four  components 
of  the  learning  style  inventory  with  ho  significant  results. 
Descriptive  statistics,  were  prepared  for  all  the  database  files 
describing  the  data  collected  on  Various  forms. 


Q#30yQ#31^Q#32  Numerous  tables  were  prepared  crosstabulating 

evaluaticn  by  reduced  variables  which  we  call  Xy  .^rhich 
are  the  flags  indicating  whether  a  variable  is  above  or  below  its 
outpoint. 

(2#33  Scatter  pilots  were  prepared  for  context  variables  by 

question  number  with  ho  significant  results. 


Q#34  Descriptive  statistics  were  prepared  for  all  variables 

by  questions. 

Q#35  Crosstabulation  was  formed  for  variables  established  by 

Judge  1  with  variables  established  by  Judge  2. 
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SUMMARY  OF  THE  PROGRAM  PRINTOUTS  FROM  WHICH  SPECIFIC  CONCLUSIONS 
HAVE  BEEN  DERIVED 


Table  16-3.   TQj2 

Table  16-2  Q21 

Table  16-3  T0700T 

Table  16-4  TBToat 

Table  16-5  TQ720eT 

Table  16-6  QD12 

Table  16-7a  TEFKO07 

Table  16-7b   T07IN 

Table  16-8  QOll 

Table^€-5  QIN^  QOUT 

Table  16-11  013 


References  to  the  SMDP  manual. 

The  page  references  for  the  particular  tests  used  are  as 
follows : 

Data  Description:  PID,  P2D,  P4b  PP  73-92 
Two-way  analysis:  P7b  PP  93-122 

Linear  Regression:  p1r;P2R  pp  235-263 

Two-way  measures  assciath  P4P      pp  143-206 
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Table  16-14 


EXftMPEE  CRbSS-RftTid  DATA  (N  unique  items,  outside=3689 ) 

The  data  presented  throughout  this  report  are  based  on  a 

division  of  unique  items  into  those  retrieved  by  project  searchers 

and  those  retrieved  by  outside  searchers*  in  an  alternative  

analysis r  the  items  retrieved  by  outside  searchers  are  counted 
unique  if  no  other  outside  searcher  found  them i   (This  data  was 
heeded^ to  analyze  the  influence  of  multiple  retrieval i ) 
Comparison  of  these  data  with  those  reported  elsewhere  in  the 
report  shows  that  the  choice  of  which  data  file  is  analyzed  does 
not  alter  the  key  conclusions.  The  value  of  the  odds 
ratio  may  change  by  a  few  percent,  according  to  which  file  is 
used  to  analyze  the  data. 


Table 

Content 

Odd 

LN(Odd 

ASEl 

T-VALUE 

# 

Ratio 

Ratio) 

_ 

1 

EVat 

X 

QVA 

2.726 

1.663 

6.668 

15;230 

2 

EVAL 

X 

QVB 

2.175 

6.777 

0.067 

11.787 

3 

EVAL 

X 

QVC 

0.232 

-1.462 

0.070 

-22.198 

4 

EVAL 

X 

QVD 

1.365 

6.311 

0.089 

3. 510 

5 

EVAL 

X 

QVE 

1.426 

6.355 

0.067 

5.307 

6 

EVAL 

X 

QVF 

1.426 

6.355 

0.067 

5.307 

7 

EVAL 

X 

UVA 

0.918 

-6.086 

0.066 

-1.312 

8 

EVAL 

X 

UVB 

1.287 

6.252 

0.080 

3.151 

9 

eVal 

X 

ove 

2.036 

6.711 

0.069 

16.512 

16 

EVAL 

X 

evb 

1.839 

6.669 

0.067 

9.186 

11 

EVAL 

X 

UVE 

1.732 

6 . 549 

0.067 

8.236 

12 

EVAL 

X 

PREBRD 

3i254 

lilSO 

6.153 

8.764 

13 

EVAL 

X 

APPLN 

lil03 

6.098 

6i676 

1.292 

14 

EVAL 

X 

LANG 

B.597 

-6.516 

6i668 

-7;640 

15 

EVAL 

X 

YEARdEN 

1.225 

6.263 

0.075 

2.768 

16 

EVAL 

X 

YEARS? 1 

1.225 

6.203 

0.075 

2.768 

17 

EVAL 

X 

YEARSP2 

1.225 

0.203 

D.075 

2.708 

18 

EVAL 

X 

SUGDBl 

1.276 

6.244 

0,096 

2.566 

19 

EVAL 

X 

SUGDB2 

1.565 

6.448 

0.110 

4.165 

20 

EVAL 

X 

SUGDB3 

1.603 

6.472 

0.131 

3.693 

21 

EVAL 

X 

PROBDEFl 

1.172 

6.159 

0.067 

2.375 

22 

EVAL 

X 

iNTENTl 

0.742 

-6.299 

0.066 

-4.512 

23 

EVAL 

X 

PROBKNOl 

2.102 

6.743 

0.068 

11.139 

24 

EVAL 

X 

iNTKNbi 

1.089 

0.685 

0.066 

1.283 

25 

EVAL 

X 

PR0BDEF2 

26 

EVAL 

X 

INTENT2 

27 

EVAL 

X 

PR0BKN02 

1.112 

6.166 

0i077 

1.376 

28 

EVAL 

X 

INTKN02 

0.969 

-6.695 

6.667 

-1.426 

29 

EVAL 

X 

CATEGORY 

1.111 

6.165 

0.067 

1.579 

30 

EVAL 

X 

CLTYSEM 

1.027 

6.027 

0.078 

0.344 

31 

EVAL 

X 

CLTYSYN 

1.436 

6.362 

0.084 

4.345 

172 


32 

EVAL 

X 

CLTyA\^__ 

1. 150 

0  i  140 

0  -.  068 

2  06^ 

33 

EVAL 

X  • 

SPECQURY 

0.  433 

-0  S37 

0-  074 

XX . / / u 

34 

EVAL 

X 

SPEC SUB J 

0.  747 

-0  299 

35 

EVAL 

X 

SPECMEAN 

0  514 

-0  666 

0  069 

-9  767 

36 

EVAIi 

y 

0  nK7 

37 

EVAL 

V 

1  606 

0  474 

0  079 

38 

EVAL 

X 

TRANSSER 

d  796 

-0  '590 

0    1  59 

—  9  0*^ 

39 

EVAL 

FINAL 

40 

EVAL 

Cl  V  *XAJ 

V 

0  647 

-0  4^6 

0  067 

—  6  5A^ 

41 

EVAL 

V 

A 

CLTYSEM2 

0  790 

—  0  '^99 

0  075 

—4   41  0 

EVAL 

A 

^4J  X  X  W  X  1^  ^ 

0  0fl9 

0  117 
U  •  XX  / 

43 

EVAL 

y 

A 

X  Xf^  V^7*> 

0  774 

-0  256 

0  074 

—  ^  470 

44 

EVAL 

SPEGORYS 

1  090 

0  0fl6 

U  •  I/O  W 

0  066 

1  "^Ofi 
X  .  J  u  o 

45 

EVAL 

y 

A 

SPEPSHS9 

0   94  9 

U  •  7X^ 

—  0  099 

0  0fi7 

—x . ooo 

46 

EVAL 

V 
A 

SPEGMENP 

1  172 

X  •  Xy  ^ 

0  -  159 

U  •  X^  Z7 

0  06^ 

9   A4  0 
2  .  4X u 

47 

EVAL 

X 

SEBeDNC2 

0.611 

-0.493 

0.067 

-7.457 

48 

EVAL 

X 

C0NSTR2 

1.279 

0.246 

0.069 

3.586 

49 

EVAL 

X 

TRANSER2 

1.640 

0.495 

0.254 

2.023 

50 

EVAL 

X 

PINAL2 

Not  applicable 
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17.  CONCLUSIONS 


17 •!  Overview 


This  study  addresses  human  aspects  and  decisions  In  information 
retrieval.    The  aim  of  the  study  is  to  contribute  to  a  foarmal  or 
scientific  characterizatibn  of  the  elements  involved  in  information 
seeking  and  retrieving,  particularly  in  relation  to  cognitive 
decisions  and  human-system  interaction. 

 The  objectives  were  to  conduct  a  series  of  observations  or 

experiments  under  as  real-life  conditions  as  possible  on  the 
following  entities  or  classes  of  variables: 

Users  who  have  questions  for  infomnation  retrieval  systems 
and  the  context  of  their  questions 

2m  Questions ^  structure  and  classification 

3.  Searchers ,  cognitive  traits  and  online  experience 

4.  Searches ,  effectiveness,  efficiency  characteristics »  and 
overlap 

5.  Items  retrieved,  distribution  and  the  variables  that  affect 
retrieval  of  items  judged  relevant  by  users 

Involved  were: 

•••  40  questions  posed  by  40  users 

•••36  outside  searchers,  3  project  (staff)  searchers,  and  1 
additional  analyst  for  question  classification  for  a  total  of  40 
searchers 

•      9  searches  fbir  each  question  for  a  total  of  360  searches 

8956  retrieved  items  for  all  of  the  360  searches 
(including  duplicates) 

54ii  retrieved  items  for  ail  searches  after  duplicates  were 
eliminated 

•••  these  5411  unique  retrieved  items  evaluated  by  users  for 
relevance 

• • •  an  assortment  of  measures  and  indicators  pertaining  to  every 
one  of  the  five  entities  ennumerated  above  and  resulting  in: 

a  total  of  90  variables  used  in  some  way  or  another  in  the 
analysis • 
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The  study  Involved  a  large  amount  of  data  and  a  long  lisiE  of 
statistical  correlations  and _ tests.    Many  of  these,  as  expected,  did 
hot  find  st-^.tistically  significant  results  or_  explahatlbhs.    In  the 
presentation  of  results  we  have  highlighted  the  quantified 
characteristics  of  the_ ehhumerated  entities  and  presented  details  of 
a  number  of  statistical  analysis  concentrating  on  those  producing 
significant  and  important  results* 

In  the  process,  a  number  of  specific  conclusions  were  made 
drawing  from  data  in  the  accompanying  tables i    In  this  chapter  we  are 
translating  the  specific  conclusions  into  a  series  of  more  general 
statements.    The  specific  conclusions  niade  in  the  chapters  on  results 
(Chapters  8  through  16)  are  extracted  (with  references  to  appropriate 
tables)  and  placed  as  supportive  evidence  under  the  general 
statements.    In  this  way,  this  concluding  chapter  stand  on  its  own 
summarizing  the  whole  study. 

There  is,  of  course^  a  llmit  to  our  cbncluslbhs.  _  As  mehtiptied 
in  the  Preface,  we  cannot  claim  generalizations  beyond  our  sample, 
any  more  than  any  other  similar  study  has  been  able  to  claim.  Still, 
we  are  offering  these  general  conclusions  to  be  taken  with  all  due 
caution  for  discussion,  confirmations,  refutation  or  as  hypotheses 
for  further  and  in-depth  studies. 

17 ml  Users  and  Question  Context 


1.    Of  the  four  characteristics  in  the  context  of  user  questions: 
(i)  problem  definition  (how  clearly  was  it  defined);  iil)  intent  (how 
specific  was  the  purpose  of  use);  (ill)  internal  knowledge  (about  the 
problem  at  hahd)j  and  _(lv)  public  knowledge  estimate  (does 
information  on  the  problem  exist),  every  variable  except  internal 
knowledge  showed  a  significant  impact  oh  the  chance  of  retrieval  of 
relevant  items.     (Relevant  and  partially  relevant  items  were 
considered  together  and  labeled  'relevant*  in  all  the  analyses  except 
onei)    The  better  the  problem  was  defined,  the  better  the  chance  that 
retrieved  items  were  relevant i    ^^less  specific  the  Intent,  the 
lesser  the  chance  that  a  retrieved  item  was  relevant^    When  estimates 
of  the  existence  of  public  knowledge  were  high,  the  chance  that  a 
retrieved  item  was  relevant  was  also  very  highi 

...  An  item  retrieved  in  response  to  a  question  for  which  the 
underlying  problem  was  considered  well  defined  (above  the  cut 
point  of  3.67  on  a  5  point  scale)  was  17%  more  likely  to  be 
judged  relevant  (Table  16-3) 

...  Ah  item  retrieved  for  a  qiiestibh  for  which  4htent  for  use  of 
the  information  was  narrowly  defined  (below  2 i 93  on  a  5  point 
scale)  was  27%  less  likely  to  be  judged  relevant  (Table-lo^) 

m mm  kn  item  retrieved  for  a  question  for  which  the  the  user 
believed  that  there  was  substantial  public  knowledge  in 
existence  about  the  problem  (above  the  cut  point  of  3.63  on  a  5 
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point  scale)  was  110%  more  likely  to  be  judged  releyaht  than  an 
item  retrieved  for  a  question  for  which  public  knowledge  was 
jiadged  to  exist  to  a  lesser  extent  (below  the  cut  point)  (Table 
16-3) 


17.3  User-Congtraint 8- Qg- Questions 

2i    Of  the  four  conditions  users  imposed  on  the  searches  of  their 
questions^  (i)  precise  or  broad;  (ii)  type  of  application  (graduate 
study,  faculty  research,  industrial,  general);   (iii)  English  only  or 
any  language;  and  (iv)  restriction  on  years  of  publication  (last  5 
years;  no  limits;  specific  years),  only  specification  of  language  and 
years  of  publication  showed  a  significant  impact  on  the  chance  that  a 
retrieved  item  was  judged  relevant.    None  of  the  variables  had 
significant  effects  on  the  precision  of  the  retrieved  set  as  a  whole. 

...  the  chance  that  a  retrieved  item  will  be  judged  relevant  is 
reduced  by  37%  if  the  range  of  languages  is  not  restricted  to 
English  (Table  16^1) 

enhanced  by  28%  if  there  is  no^  restriction  given  on  the  year  of 
publication  (Table  16-1) 

...  the  mean  precision  for  questions  categorized  by  application 
are:  faculty  research  50%,  graduate  study  49%,  industry  49%, 
other  66%  (Table  16-2) 


...  the  mean  precision  for  questions  restricted  to  English  was 
56%  and  for  questions  not  restricted  by  language  it  was  42% 
(Table-16^2) 

•ii  the  mean  precision  for  questions  restricted  to  the  last  5 
gears  was  58%,  to  no  limit  on  years  48%,  and  to  last  15  years 
56%  (Table  16-2) 

...  the  mean  precision  for  all  questions  (i.e.,  union  of  output 
for  9  searches  for  each  question)  was  51%. 


17.4  Consistency  and  Estimate  of  Context  by  Searchers 


3.    The  judgement  between  users  and  searchers  on  the  four 
characteristics  of  question  context  findicated  on  a  five  point  scale) 
showed  substantial  cons.!stency  or  agreement  on  problem  definition, 
followed  by  intent  i    ^??™s?t  on  estimates  of  Public  knowledge  was 
low:  users  judged  it  considerably  higher.    The  users  estimate  of 
their  internal  knowledge  as  expected  was  higher  than  searchers.  Thus 
on  some  context  characteristics  searcher  estimates  of  context 
significantly  paralleled  user  estimates. 
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> » »  On  problem  definition  L5  out  of  40  cases  had  exact 
agreement;  14  of  the  remaining  cases  differed  by  one  puint  on 
the  scale  (table  14-1) 

•••Oh  intent  in  11  cases  out  of  40  there  was  exact  agreement 
and  in  25  cases  there  was  agreement  within  one  point  (table 
14-1 ) 

•••On  estimates  of  the  probability  that  Information  exists  in 
public  knowledge  exact  agreement, occurred  for  only  6  cases  and 
agreement  within  one  point  in  only  16  (6+10)  (but  of  40) 
casei3y  for  a  total  of  16  cases  of  either  exact  agreement  or 
agreement  within  one  points    The  most  common  occurrence  (8 
cases)  was  a  difference  of  2  points •    The  user's  estimate  was  2 
points  higher  than  the  searcher's  estimate  (Table-14^3^) 

i*»  On  internal  knowledge  (as  expected)  on  26  of  the  cases  the 
user's  estimate  (of  his  or  her  personal  knowledge)  was  two  or 
more  units  higher  than  the  searcher's  estimate  (Table  14-1) 

A.    Of  the  four  characteristics  of  context  of  questions  judged  by 
searchers  (as  opposed  to  users)  problem  definition  and  internal 
knowledge  had  no  sighificaht  impact,  on  the  chance  of  retrieval  of 
relevant  items,  while  intent  and  public  knowledge  had  a  small  effects 

•  ••  User's  liitent  as  assessed  by  the  searchers  showed  an 
oppdsitiB  correlation  from  the  user's  own  assessment  of  the 
chance  of  relevance  (Table- 16^3) 

•  •  Qser's  intent  as  narrowly  defined 
(above  the  cut  point  of  2i83  on  a  five  point  scale)  an  item 
retrieved  was  Sl%  more  likely  to  be  relevant  (table  16-3-) 

•••An  item  retrieved  in  response  to  a  question  for  which  a 
searcher  assessed  substantial  public  knowledge  (above  2.96)  was 
11%  more  likely  to  be  judged  relevant  than  one  retrieved  in 
response  to  a  question_on_ which  the  public  knwoledge  was  judged 
below  this  cut  point  (Table  16-3) 


17,5  insistency  of  Judgement  on  Question  Classlfi^^lon  Between  Two 
Judges 


agreement  between  two  judges  in  a  number  of  categories  used  for 
classification  of  questions:  (1)  domain  (as  expressed  by  the  number 
of  Dialindex  categories);   (ix)  clarity  of  semantics;   (lil)  clarity  of 
syntax;   (iv)  specificity  of  the  subject  of  the  question;  and  (v) 
constraints  in  the  question^    There  was  poor  agreement  on:  (1) 
.specificity  of  the  query  (about  the  subject)  of  the  question^  and 
(11)  complexity  or  number  of  concepts  in  the  question.  Overall^ 
question  classification  as  it  was  designed  seems  to  be  valid  to  a 
great  extent. 
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•  ••  On  the  estimate  of  domadta  (the  ntanber  of _l^±aiindex 
categories),  the  two  jadges  agreed  exactly  16  times  out  of  40 
and  were  within  one  unit  30  (16  +  14)  times  out  of  40  (Table 
14-2  for  this  and  all  of  the  following  conclusions) 

•••  Oh  clarity  of  semantics,  the  judges  exactly  agreed  23  times 
and  were  within  1  unit  31  times 

•  ••  Oh  clarity  of  syntax,  the  judges  exactly  agreed  29  times  arid 
were_ within  1  unit  34  times.    This  resulted  in  perfect  agreement 
on  the  average  clarity  score  (semantics  and  syntax)  in  20  cases 
and  agreement  within  one  unit  in  38  cases 


•  •  •  On  specif tcit:y^f—the^Bub4get:>  there  was  exact  agreement  in 
19  cases  and  agreement  within  one  unit  in  35  cases 

•••  With  regard  to  specificity  of  the  query  about  the  subject, 
agreement  was  not  good  at  all.  Disagreement  ranged  from  -4.75 
to  +3i7i  There  was  agreement  to  within  one  unit  in  only  17  of 
the  40  cases  •  The  mean  specificity  score  was  widely  scattered 
due  to  the  lack  of  inter-judge  agreement  on  specificity  of  the 
query 

• . •  Oh  complexity,  the  two  judges  agreed  in  only  five  cases  bh 
the  number  of  search  concepts  involved  iti  the  question,  and  diie 
of  the  Judges  estimated  the  number  of  concepts  was  greater  than 
the  number  estimated  by  the  other  judge  in  30  of  the  40  cases 

••^  There  was  substantial  agreement  on  the  number  of  

constraints:  20  cases  out  of  40^    One  of  the  judges  estimated 
the  number  of  constraints  was  greater  than  the  number  estimated 
by  the  other  judge  in  the  remaining  20  cases 

••»  In  the  study  of  the  additional  176  questions^  the  percent  of 
inter-judge  agreement  was  higher  than  on  the  40  questions  in  the 
study  but  categories  for  which  there  was  the  'most  agreement' 
and  the  'least  agreement'  were  stiil  the  same  as  described  above 
(Table  14-3) 


17^6  Impact- of  Quegtiog^Classiflcation^Categories 

6*    None  of  the  question  classification  categories  had,  at  the  same 
time,  a  significant  impact  on  the  chances  for  retrieval  of  relevant 
items  and  impact  bh  relevant  retrieval  in  the  same  direction  (same 
sign)  for  both  judges  classifying  the  questionr.    However,  tor  three 
categories  both  judgments  were  in  the  same  direction  and  there  was 
some  significance  in  effects  on  retrieval  for:  1 .  clarity  of  syntax, 
2,  number  of  constraints,  and  3,  number  of  concepts: 


•••  An  item  retrieved  in  response  to  a  question  for  which  the 
clarity  of  syntax  was  high  was  more  likely  to  be  judged  relevant 
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response  to  a  question  In  which  clarity  of 
syntax  was  low  (Table  16-4  for  this  and  the  following 
conclusions) 

•••An  Item  retrieved  in  response  to  a  question  with  a  larger 
number  of  constraints  was  more  likely  to  be  relevant  than  one 
retrieved  In  response  to  a  question  with  a  fewer  number  of 
constraints 

•••An  item  retrieved  in  response  to  a  question  with  a  greater 
number  of  coaoepts  was  more  likely  to  be  judged  relevant  than 
one  retrieved  for  a  question  with  a  lesser  number  of  concepts 

In  other  words,  clearly  structured  questions  with  a  greater 
number  of  concepts  and  constraints  significantly  favored 
retrieval  of  relevant  items • 


17^7  Searcher  Characteristics 


7.    Some  coghltlye  characteristics  (as  measured  by  the  respective 
tests)  had  positive  and  others  negative  Impact  on  relevance  of 
retrieved  items  •    Positive,  linpact  resulted  from  higher  scores  on  the 
Remote  Associates  Test  (RAT)  (relatirig  to  semantic  association) ;  the 
Abstract  Coticeptualizat ion  (AC)  learning  mode  from  the  Learning  Style 
^^Y^^^J^^y  (^SI)  (relating  to  "the  ability  to  generate  concepts  that 
integrate  observations  into  logically  sound  theories")  and  higher 
^?9???_9?_^^f_99?^^??^         score^  Abstract  Conceptualization-Concrete 
Experience,  indicating  the  extent  an  Individual  emphasizes 
abstractness  over  concreteness  in  learning  style.    Negative  impact 
resulted  from  high  scores  on  Employee  Aptitude  Survey  (EikS)  (relating 
to  deductive  Inference  ability  tested  by  solving  questions  involving 
symbolic  inequalities)  and  having  a  Concrete  Experience  (CE)  learning 
mode  from  LSI  (relating  to  "the  ability  to  Involve  oneself  fully, 
openly  and  without  reservation  in  new  experiences").    Other  learning 
modes  had  no  impacts 

•••  An  item  retrieved  by  a  searcher  who  scored  high  on  the 
Remote  Ai3SQCiates  Test  was  65%  more  likely  to  be  relevant  than 
one  retrieved  by  a  searcher  with  a  low  score  on  the  test 

•••  An  item  retrieved  by  a  searcher  ranking  high  on  the  Abstract 
Conceptualization  learning  mode  was  25%  more  iikaiy  to  be 
relevant  than  one  retrieved  by  a  searcher  ^th  a  low  score  in 
that  mode 

•••  An  item  retrieved  by  a  searcher  with  a  high  value  in  the 
composite  learning  style  score  called  AC  minus  CE  was  27^5%  more 
likely  to  be  relevant  than  one  retrieved  by  a  searcher  with  a 
low  score  on  that  style 

•••A  searcher  with  a  high  Concrete  Experience  score  had 
decreased  chances  of  retrieval  cf  relevant  items  by  27% 
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iii  There  was  a  marginally  significant  iridicatibn_ that  ah  item 
retrieved  by  a  searcher  with  a  high  score  oh  the  Employee 
Aptitude -Survey  was  11%  less  likely  to  be  relevant  than  an  item 
retrieved  by  a  searcher  with  a  low  score 

Thus,  if  these  cognitive  tests  measure  abilities  which  they 
claim  to  measure,  a  searcher  with  higher  verbal  abilities  and 
preferring  ah  abstract  style  of  learning  has  increased  chances 
for  retrieving  relevant  items  and  a  searcher  with  high 
mathematical/logical  abilities  and  preferring  a  learning  style 
based  on  concrete  experiences  has  decreased  chances  of 
retrieving  relevant  items. 

8.    Frequency  of  DIALOG  searching  had  no  impact  on  the  chance  that 
retrieved  items  were  relevant. 

•  •  DIALOG  daily  or  twice  a  week  did  not 

differ  in  retrieval  of  relevant  items  from  searchers  who  search 
DIALOG  once  a  week,  twice  a  month,  or  less. 


17.8  Search  Efficiency  Characteristics 

9.    Of  the  six  measures  characterizing  the  efficiency  of  the  search 
(number  of  cdtmnahds  used:  number  of  command  cycles  used  from 
selecting  terms  to  displaying  results;  number  of  search  terms  used; 
online  connect  time;  preparation  time;  and  total  time)  the  most 

^^P^^^^^^^^creasing^  of  retrieving  relevant 

items  was  command  cycles  used.    Negative  impact  resulted  from  higher 
numbers  9^  search  terras  and  more  total  time  spent  on  the  search. 
Other  factors  were  not  significant.     (Two  types  of  searches  were 
involved:  outside  searches^  searchers  searching  each 

assigned  question  once,  and  project  searches  searching  each  question 
four  times  varying  methods  for  each  search  as  explained  in  Section 
17.9). 

...  items  retrieved  by  200  outside  searches  with  a  large  number 
of  cycles  were  25%  more  likely  to  be  relevant  than  those 
retrieved  by  searches  employing  a  lesser  number  of  cycles  (Table 
i6^7a) .    Similarly,  for  160  project  searches,  a  greater  number 
of  cycles  produced  a  21%  increase  in  the  chance  of  relevance 
(Table^  16-7}x) 

•••  Items  retrieved  by  outside  searches  which  took  a  greater 
amount  of  preparation  time  were  13%  less  likely  to  be  relevant 
(Table  16"7a) 

...  The  use  of  greater  amount  of  total  time  produced  a  36% 
decrease  in  the  chance  that  a  retrieved  item  was  relevant  for 
project  searches  (Table  16-7b) 

...-The  use  of  greater  numbers  of  search  terms  produced  a  39% 
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decrease  ±n  the  chance  of  relevance  for  project  searches  (Table 
16-7b) 


•••  On  the  average,  per  search^  there  were:  15  commands,  3 
cycles,  10  search  terms,  10  minutes  preparation  time^  12  minutes 
online  time^  and  22  minutes  total  time  used.    However,  the 
ranges  varied  widely  and  the  distributions  were  either  skewed  In 
one  direction  or  had  several  peaks_and  therefore  mean  values 
must  be  interpreted  with  caution  (Table  10-3) 

None  of  the  effectiveness  characteristics  of  a  search  (total 
number  of  items  retrieved,  search  recall,  and  search  precision) 
had  more  than  5%  of  its  observed  variation  explained  by  the 
efficiency  characteristics  (number  of  commands,  cycles,  and 
search  terms,  and  amount  of  online,  preparation,  and  total  time) 
(Section  16. A. I) 


17.9  Types  of  Searches 


1P*_  There  was  significant  difference  in  recall  but  no _sighif leant 
difference  in  precision  for  the  four  different  types  of  searches  (so 
called  project  searches)  based  on:  1.  taped  interview  with  user  on 
the  problem  underlying  the  question;  2.  problem  interview  plus 
written  question;  3.  terms  from  the  question  only  (no  elaboration, 
'dumb'  search) I  and  4,  terms  from  the  question  plus  elaboration 
through  a  thesaurus. 

Search  type  i  (problem  tnterview)  had  the  highest  mean 
recall  at  32%,  followed  by  search  type  4  (question  and 
thesaurus )  at  25%,  then  by  search  type  2  (problem  and  question) 
at  23%^  and  finally  search  type  3  ('dumb'  searchj  at  18%.  The 
distribution  is  skewed  toward  the  lower  end  of  the  scale  (Table 
16-10) 

The  mean  precision  was  similar  for  all  four  types  of 
searches:  for  type  1  and  2  it  was  63%,  for  type  4,  61%  and  for 
Type  3,  56%.    Values  are  skewed  toward  the  upper  end  (Table 
16^10) 

The  best  performance  tn  recall  and  precision  was  by  searches 
done  on  the  basis  of  verbal  statements  obtained  by  interviewing 
the  user  who  elaborated  on  the  problem  at  hand  and  his  or  her 
intent  for  using  the  information.    Performance  was  worst  when 
v7ords  from  the  written  questions  were  used  alone  without 
elaboration  as  if  they  had  been  picked  automatically, 

11.    Recall  and  precision  for  project  searches  averaged  slightly 
higher  than  for  outside  searches.    The  project  searches  had  access  to 
users'  problem  statements,  the  outside  searches  did  not.    Both  had 
access  to  the  written  questions  and  the  same  thesauri  for 
elaboration. 
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Mean  search  recall  for  project  searches  was  25%  and  for 
outside  searches  It  was  20%,  mean  precision  for  project  searchies 
was  61%  and  for  outside  searches,  54%  (Table  16-9),  This 
underscores  the  absence  of  an  Inverse  relation  between  precision 
and  recall  as  Is  discussed  In  Section  17 •12. 


a*  ^?  ?^?_^Y?F^??i  9^??^^®  searches  used  more  commands  per 
search  than  project  searches  (16  vs.  13),  more  cycles  (4  vsi  3), 
the  same  numSer  of  search  tc»rms  (10),  more  preparation  time  (15 
mln.  vs.  11  min.),  more  online  time  (16  min,  vsi  12  mln.)  and 
more  total  time  (31  mln.  vs.  23  mln.)  (Table  16-9) 


17.10  Overlap  of  Search  Terms  and  Items  Retrieved 

[Note:  Each  question  was  searched  by  five  searches  by  different 
searchers,  called  'outside  searches'.    The  question  was  also  searched 
by  four  prcject  searches  (see  preceeding  section).    Overlap  was 
studied  only  for  the  five  otitslde  searches,  because  they  were  all 
done  on  the  same  basis:  the  written  question  statement  only.] 

12 •    Tliere  was  surprisingly  low  agreement  on  selection  of  search 
terms  (on  the  same  basis  of  the  written  question  from  the  user) 
between  different  searchers  in  searching  the  same  question. 

...  The  mean  agreement  between. searchers  on  search  terms  for  the 
same  question  was  27%  (Table  _12""1 ) .    Hbweyerj  the  distribution 
was  very  much  skewed  toward  the  low  end  (Table  15-1). 

...  Iii  11%  of  the  cases  there  was  less  than  5%  agreement  oti 

search  terms.    Iii  20%  of  the  cases  agreement  was  10%  or  less» 

In  44%  of  the  cases  agreement  was  20%  or  less.    In  56%  of  the 

cases,  agreement  was  25%  or  less#  (Table-15^) 

13 i    The  overlap  of  retrieved  items  between  different  searchers 
searching  the  same  question  (on  the  same  basis  of  written  question 
statement)  was  also  surprisingly  low  and  it  was  even  lower  than  the 
degree  of  agreement  observed  In  selection  of  search  terms.    This  was 
true  In  retrieval  of  both  all  Items  and  retrieval  of  relevant  Items, 
however,  the  overlap  in  relevant  items  was  slightly  greater. 

...The  mean  overlap  of  items  retrieved  for  the  same  question  by 
different  searchers  was  16%  for  all  Items  retrieved  and  18%  for 
relevant  Items  retrieved  (T*eMe  l2^44 ) .    However^  the 
distribution  is  skewed  toward  the  low  end.     (Ta^le-15^2  and 
15-^3) 


...  In  retrieval  of  ail' Items  (relevant  and  nonrelevant).  In  59% 
of  the  cases  overlap  was  less  than  5%  and  in  66%  of  the  cases 
overlap  was  no  more  than  10%  (i.e.,  10%  or  less).     (Table  15-2) 

..e  In  retrieval  of  relevant  items,  in  59%  of  the  cases  overlap 
was  less  than  5%  and  in  64%  of  the  cases  overlap  was  10%  or 

182 


203 


less.  (Table-i5^) 


•ii  The  low  degree  of  overlap  in  selection  of  search  terms  and 
retrieved  items  was  one  of  the  most  surprising  findings  of  the 
study. 

14 i    Surprisingly,  the  degree  of  agreement  in  search  terms  and  the 
degree  of  agreement  initems  retrieved  tor  searches  of  the  same 
question  are  not  significantly  related. 

...  The  Substantial  disagreement  in  items  retrieved  does  not 
find  its  explanatlph  in  the  much  lower*  but  still  significant* 
disagreement  in  selection  of  search  terms.    We  found  no 
significant  relation  between  the  disagreement  measured  by  search 
teiTTis  and  the  disagreement  measured  by  overlap  in  itemis 
retrieved.    Only  2.5%  of  the  variation  in  overlap  of  retrieved 
items  can  be  attributed  to  overlap  in  search  terms.  (Table 
15^5) 

. .    This  is  another  one  of  the  surprising  and  significant 
findings • 


17.11  Odds  of  Relevance  in  Retrieval  of  Duplicate  Items 

15.    As  explained,  a  question  was  searched  by  five  searches  by 
different  searchers  arid  the  decree  of  agreement  of  all  items 
retrieved  was  low.    However,  when  an  item  was  retrieved  by  more  than 
one  searcher  (2,  3,  4  or  5  times)  it  had  a  surprisingly  greater 
chance  of  being  relevant  (as  opposed  to  nonrelevant).    The  more  often 
the  same  item  was  retrieved  by  different  8earghgs^£or-t:he-saTfle 
question^  the_more  -likely^  It— wag- to-be  relevant^ 

timportant  note:  In  ail  previous  and  subsequent  analysis,  items 
judged  by  users  as  Relevant  or  Partially  Relevant  were  treated 
together  (R  +  pR),  but  for  this  analysis  (to  sharpen  the  view)  only 
those  items  judged  R*  Relevant,  were  considered  and  pR,  Partially 
Relevant*  items  were  disregarded.] 

...  For  all  retrievals  (those  items  retrieved  once*  twice,  three 
times  or  more)  the  odds  that  the  item  was  relevant  as  opposed  to 
nonrelevattt  was  5  to  10.     (Tables  15^4^1^  and  45^4^2^  for  these 
and  subsequent  conclusionis) 

••i_?9?       item  retrieved  only  once  (in  five  searches)  the 
corresponding  odds  were  4  to  10 

...  For  items  retrieved  twice  (in  five  searches)  the  odds  were 
even^  10  to  10 

...  For  items  retrieved  three  or  more  times  (i.e.,  three*  four* 
or  five  times)  the  odds  that  the  item  was  relevant  were  16  to 
10.    In  other  words*  when  an  item  is  retrieved  three  or  more 
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times  the  odds  favoring  relevance  improve  considerably. 

•      We  consider  thiis  the  most  significant  finding  of  the  study. 


17«i2  Recall  and  Precision 


16 i    The  mean  search  recall  and  precision  values  found  confirm 
similar  figures  from  many  other  studies,  however,  considering  the 
means  alone  is  misleading  because  the  distributions  were  skewed  in 
one  direction, 

the  mean  recall  forall  360  searches  was  about  22%  and  the 
mean  precision  about  57%  (Tables  12''1 ) 

17,    In  this  study,  when  recall  was  plotted  against  precision  the  two 
were  not  inversely  related  a  is  they  aire  widely  considered  arid  as  the 
so  called  Cleverddn  Law  stated*    To  the  contrary,  wheri  either  recall 
or  precision  was  considered  the  independent  variable,  the  other  rose 
slightly.    As  recall  rose  so  did  precision,  but  rather  mildly;  or 
vice  versa,  as  precision  rose  do  did  recall,  also  mildly. 

• . •  the  plot  of  recall  and  precision  showed  a  large  amount  of 
scatter  and  a  direct  linear  relation  (Figure  12-1) 

18«    A  low  percentage  of  the  variation  observed  in  recall  and  in 
precision  was  explained  by  the  variables  used  In  this  study.    For  the 
most  part^  we  still  do  hot  know  what  variables  have  the  greatest 
effect  on  recall  and  precision. 

...  The  moisit  important  explanatory  variable  for  precisibri  was 
the  user's  estimate  of  public  knowledge  (i.e.,  the  probability 
^f  P'^^^^^ted  by  the  user  that  information  about  the  ^p^  at 
hand  existed  in  public  knowledge).    This  explained  10%  of  the 
observed  variances    The  next  variable  which  ex^  5%  of  the 

variance  in  precision  was  the  searcher  characteristic  measured 
in  this  study  by  the  Remote  Associates  Test  designed  to  test 
word  association  ability*    Together  these  two  variables 
explained  15%  of  the  variation  in  precision^    No  other  variable 
from  among  thosevariables  studied  passed  the  test  of 
significance.     (Table  16-6) 

...  The  possibility  for  explairiirig  recall  was  substantially 
worse.    Only  one  variable  proved  mildly ^significant :  the 
cognitive  score  of  searchers  on  the  Learning  Style  Inventory 
(AG  -  GE)  which  identifies  a  style  of  learning  in  which  an 
individual  emphsizes  abstractness  over  concreteness .  It 
explained  somewhat  less  than  5%  of  the  observed  variation  in 
recall.    No  other  possible  variable  passed  the  significance 
test.     (Table  16-6  and  16-8) 

...  In  ho  case  was  more  than  2.5%  of  the  variance  observed  in 
precision  and  recall  explained  by  the  efficiency  variables 
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(l«e«^  number  of. search  termsj  number  of  commands ,  number  of 
cycles ^  preparation  time^  bniirie  time,  and  total  titae)j  that  is, 
by  the  same  variables  that  had  an  impact  oil  relevance  (see 
Conclusion  number  9).    This  was  because  the  liavel  of  analysis 
possible  for  the  search  ais  a  whole  (the  search-wise  level) 
involving  precision  and  recall  was  not  sensitive  enough  to 
detect  impact  as  did  the  level  of  analysis  used  for  the 
relevance  of  each  item  retrieved  (the  item-wise  level).  (Table 
16^) 


17il3  Utility  Assessment  by  Users 

19.    The  following  four  types  of  utility  assessments  by  the  users  (of 
the  five. types  of  assessments  used  for  the  study)  correlated 
positively  with  the  chance  that  a  retrieved  item  was  relevant:  (i) 
dollar  value  assigned  to  items  received,  (ii)  overall  worth  of  the 
information  received  in  terms  of  time  spent,  (iii)  contribution  to 
resolution  of  the  problem  at  hand,  and  (iv)  overall  satisfaction. 
The  fifth  measure  (time  spent  evaluating  the  items  received)  had  no 


items  retrieved  in  searches  of  questions  for  which  the  user 
considered  the  value  of  the  information  received  above  the 
cutpbiht  of  $75  were  28%  more  likely  to  be  relevant  than  items 
retrieved  in  searches  valued  below  $75  (Table  16-4  for  this  and 
following  conclusions) 


...  items  retrieved  in  searches  of  questidnis  for  which  the  user 
assigned  the  worth  of  information  received  (in  tertns  of  the  time 
he  or  she  spent  participating  in  the  study)  above  the  cutpoint 
of  four  (on  a  five*' point  scalfe) ,  i.e. ,  the  results  were  worth 
more  than  the  time  spent,  were  164%  more  likely  to  be  relevant 

•••  items  retrieved  in  searches  of  questions  for  which  users 
considered  the  information  contributed  to  the  resolution  of  the 
problem  at  hand  (cutpoint  of  3.15  on  a  five  point  scale)  were 
84%  more  likely  to  be  relevant 

...  items  retrieved  in  searches  of  questions  for  which  users 
indicated  ahigh  degree  of  satisfacti^m  (above  the  cutpoint  of 
3.62)  were  73%  more  likely  to  be  relevant 

...  in  other  words,  relevance  of  items  submitted  and  user 
utility  scores  paralleled  each  other. 

17.14  On  the  Approaches  to  Statistical  Analysis 

20.    Two  levels  of  statistical  analysis  were  used  in  this  study: 
search--wlse  and  item-^ise.    On  the  search-wise  level,  explanations 
were  sought  for  the  impact  of  given  variables  on  precision  and 


effect • 
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recall,  that  is  on  the  effectiveness  of  the  searches  as  a  whole.  On 
the  item-wise  level,  explanations  were  sought  for  relevance  of  items 
retrieved  on  an  item  by  item  basis.    The  item^iise  level  proved  much 
more  powerful  in  providing  explanations, and  insight.    Results  were 
obtained  from  the  Item-wise  leveled,  analysis  which  the  search-wise 
level  analysis  could  not  provide.    While  the  item-wise  approach  to 
analysis  is  similar  to  analysis  techniques  which  are  common  in 
biomedicine^  to  pur  knowledge  it  has  not  been  applied  in  information 
science  until  this  study.    The  results  obtained  here  recommend  its 
use.    it  seems  that  the  notion  of  precision  and  recall  of  a  search  as 
a  whole  when  used  in  connection  with  explanatory  variables  is  in  need 
of  reconsideration. 
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18.  DISCUSSION 


The  tnalti  orientation  of  the  study  was  collection  of  data  on  what 
lis  happening  In  the  realtii  of  searching  for  Information,  particularly 
online  searching;    It  was  an  exploratory  study  because  very  little 

^^?^_?^ft  on  the  variables  studied*        .P^?®^'^®^.?????^??*  ?® 

not  study  why  the  things  we  observed  happened i    In  that  regard,  we 
?^?_?P??^?-?^?  f-^^^S  ^ith  everybody  else.    Our  remarks  here  are  not 
directed  at  auch  speculations,  but  they  do  attempt  to  iterate  certain 
implications • 

The  model  of  information  seeking^  concentrating  on  the  context 
of  questions  and  problem  orientation^  worked  well  in  that  it  showed 
an  Impiact  on  relevance.    The  model  heeds  sharpening  no  doubt,  but 
even  so,  it  provides  several  Impllcatlphs  for  design  of  infdnnatibn 
systems  arid  for  the  practice  of  searchlrig.    Both  have  to  consider 
giving  users  the  charice  to  express  the  problem  at  hand  at  some  length 
together  with  an  opportunity  for  users  to  give  their  estimate  of  what 
information  most  likely  exists  on  the  problem.    This  information  can 
then  be  Incorporated  In  searching; 


In  this  study,  as  in  many  other  studies  involving  searchers  and 
searching,  one  set  of  conclusions  is  negative.    We  uncovered  more 
evidence  of  things  which  do  not  affect  retrieval  of  relevant  Items  or 
precision  and  recall  than  evidence  of  things  which  do.    There  is  an 
absjence  of  any  clear  influence  of  particular  variables  (be  they  user  ^ 
question^  searcher^  or  search  related)  oh  the  relevance  of  retrieved 
items  ahd  even  less  of  ah  Indication  of  variables  affectlhg  recall 
ahd_ precision.    What  affects  the  retrieval  of  relevaht  items  can  be 
explalried  orily  to  a  certalri  arid  rather  small  degree.    Thus,  it  is  not 
surprising  that  searching  is  still  more  of  an  art  than  an  algorithm 
and  that  prdvldlrig  intelligent  search  systems  or  modules  and  expert 
systems  in  broader  subject  areas  has  met  with  limited  success;  This 
also  implies  that  education,  training,  and  guidance  of  searchers  (be 
they  intermediary  or  end  users)  is  of  paramount  importances  However, 
one  has  to  look  very  carefully  at  the  type  of  such  education, 
training,  and  guidance. 

One  of  the  more  interesting  positive  findings  is  this:  items 
retrieved  by  searchers  with  a  high  score  oh  the  Remote  Associates 
Test^  which  is  highly  dependent  on  Idlbmatlc  Erigllsh  language  f luericy 
(without  regard  to  particular  subject  matter)  are  more  likely  to  be 
relevarit.    Fluericy  Iri  assdclatlbris  Iri  Erigllsh  arid  in  English  idioms 
seem  to  be  ari  important  characteristic  of  more  effective  searchers; 
Surprlslrigly  enough,  searchers  scoring  higher  on  a  test  of  symbolic 
reasoning  (involving  inequalities)  performed  poorer.    If  the  results 
hold  at  large  the  implications  are  obvious. 

Another  important  finding  relates  to  cycling:  items  retrieved  in 
searches  with  higher  numbers  of  search  cycles ^  (or  in  other  words  for 
searches  In  which  the  searcher  displayed. results  more  frequehtly), 
are  more  likely  to  be  relevant.    The  implication  is  that  feedback  is 
very  important.    This  supports  system  designs  Incorporating  relevance 
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feedback  in  the  search  process;    Unfortunately,  such  system  features 
while  common  in  the  laborat       have  not  as  yet  reached  existing  large 
public  and -commercial  information  systems  and  vendors; 

The  most  important  finding  is  the  very  large  differences  in 
search  terms  selected  for  searching  and  in  items  retrieved  by 
different  searchers  searching  the  same  question.    Higher  recall  is 
achieved  only  when  outputs  from  several  searchers  are  merged.  This 
implies  that  if  a  user  desires  a  search  of  higher  effectiveness  he  or 
she  should  parcel  the  search  put  to  several  searchers.    Those  items 
that  are  retrieved  in  common  by  several  searchers  have  a  much  higher 
chance  of  being  relevant.    Implications  for  design  of  automated 
search  procedures  are  obvious:  multiple  searching  of  a  question  from 
different  "angles"  provides  a  good  way  of  assuring  the  retrieval  of 
relevant  items  and  for  ranking  them. 

The  data  strongly  suggest  that  the  searcher  is  the  'weak  link' 
in  the  retrieval  of  items  from  large  files;    Searchers  may  vary 
widely  in  their  assessment  of  the  meaning  of  a  question  and  thus  in 
their  selection  of  search  terras  and  in  what  they  retrieves 

Finding  large  differences  among  searchers  has  implications  in 
these  areas: 

1.  Ejducatioh»  training^  guidance.  Questions  should  be  raised 
abblit:  the  adequacy  of  searcher  education;  the_ 
appropriateness  of  a  division  of  emphasis  which  stresses 
mechanics  of  searching  and  the  technology  arid  systems 
features  over  question  analysis  and  search  strategy;  the 
nature  of  existing  system  search  features  and  guidance 
features. 

2.  Research i  There  is  a  clear  need  for  a  better  understanding  of 
the  nature  of  observed  differences  and  for  confirmation  of 
the  results.    Without  such  an  understanding,  design  of 
intelligent  search  systems  and  features  becomes  a  guessing 
game  based  on  speculation. 


3.  Design.  Information  systems  need  features  that  will  minircize 
the  differerices  in  searchirig,  such  as  help  iri  question 
processing,  automatic  question  trarisf oxrmatibri,  focused 
human-system  interfaces,  and  relevance  guidance,  feedback  and 
estimates. 

A  significant  amount  of  research  on  users,  questions,  searchers, 
and  searching  is  still  needed  to  lift  searching  from  an  art  ot  a 
science  and  to  take  the  guesswork  out  of  the  process  of  designing 
htanan-system  interfaces  involving  searching.    To  be  successful, 
intelligent  systems  desired  for  the  future  are  riot  possible  without 
such  research.    We  know  a  good  deal  less  than  we  think. 
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iNTROCDTION  TO  THE  APPENDICES 


 ?^??®_?PP?^^^?6?_??'e  a  companion  to  the  final  report  of  an  NSP 

sponsored  study  concictintrating  on^a  characterization  of  the  elements 
in  information  seeking  and  information  retriev£ng>  partlcuierly  in 
relation  to  the  cognitive  decisions  s\rd  human  interactions  involved 
in  online  information  retrieval •    The  objectives  were  to  conduct  a 
series  of  experiments  related  to  the: 

1.  Ccntgxt^  of  questions  as  provided  by  the  users 

2.  Structure  and  classification  of  questions 

3i  Cognitive  traits  ari;d  decietcn  making  of  searchera 

4.  Comparative  nature  of  the  search  by  different  searchers  of 
the  same  question 

One  question  was  submitted  by  each  of  the  40  users  partlcl^ 
in  the  project^    the  users  also  indicated  the  context  and  constraints 

95_^^®_5'^®®^i9?_??^_¥®??_i?^???^^^?^  about  the  problem  at  hand i  A 
S^^np  of  4b  searchers  was  assCTbledi    The  searchers  were  tsted  on 
three  cognitive  tests  and  provided  d^^  online  experience* 

the  project.  _The_5erged  or  union  set  ?fter  elimination  of  duplicates 
for  ail  nine  searches  for  a  question  was  sent  to  a  user  for 
valuation  of  relevance  snd  utility,    for  ail  360  searches  8956  items 

^'^^iy^^^.^^^^i^^?^  9?  _^^®®^^'?8_^^?_^?®9^9^9y_?^i_^l?^?l§^?^^'^_9^_'^ 
variables  involvedand  a  correlation  for  significance  for  ail  of  the 

90  variables  that  make  sense  to  correlate* 

A  tremendous  amount  of  *raw'  and  analyzed  data  was  generated  in 
the  projects  The  aim  of  the  final  report  has  been  to  Present,  in  as 
much  detail  as  possible »  the  data  and  results  of  the  study i    to  this 

?'^^_?^f_ii?*^?_'^fP^^^_i?_P'^^^?9®^_i'^  ?^9  P??!®*  first  consists  of 

a  presentation  of  methods »  results »  rtnd  conclusions o    This  second 
part  is  composed  of  a  set  of  appendices  containing  as  much  of  the 
^raw*  data  as  ve  could  reasonably  reproduce i    The  rest  of  the  data  is 
deposited  in  machine-readabiefora  with  the  complete  project  archives 
at  two  universities:  Case  Western  Reserve  University  (contact  Paul 
Kantor)  and  Rutgers,  the  State  University  of  New  Jersey  (contact 
tefko  Saracevic)i 

the  study  produced  a  wealth  of  data  that  in  itself  is  a 
considerable  and  even  unique  resource*    Our  idea  is  to  exploit 
further  this  data  and  to  provide  open  access  to  this  data  to  all  who 
desire  to  use  it  further^  and>  of  course  have  resources  to  do  so. 

the  first  part  of  the  report  stands  on  its  own.    The  appendices 
are  an  invitation  for  verification,  refutation,  replication,  and 
further  in-depth  study. 
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APPENDIX  A 


USER  QUESTIONS  FOR  ONLINE  SEARCHES  AND  CONTEXT  MEASURES  FOR 

QUESTIONS  QOOl  TO  QOAO 


APPENDIX  A.  USER  QUESTIOHS  FOR  ONLINE  SEARCHES  AND  CONTEXT  ItffiASURES 
FOR  QUESTIONS  QOOl  TO  Q040 


_  _  For  each  of  the  40  questions  used  In  the  project  a  copy  of  Form 
15  (see  Appendix  H)  was  cbiipleted  to  include  the  following: 

1«  Question  number  assigned  for  the  project  (I.e.,  QOOl  through 
Q040) 

2.  DIALOG  database  file  used  (including  the  DIMiOG  file  iiumber 
and  the  DIALOG  file  name) 

3.  Searcher  code  nimibers  for  the  five  outside  and  four  project 
searches  (The  project  search  codes  are  explained  in  detail  In 
the  first  volume  of  this  report^  Table  9-1 Q 

4.  Brief  title  transcribed  as  stibmitted  by  the  user 

5.  Question  statement  transcribed  as  submitted  by  the  user 

6.  Type  of  search  requested  by  the  user 

7o  Context  scales  (1*  prbbleS  definition,  2*  intent^  3*  public 
knowledge,  and  4.  internal  knowledge).    The  user  completed 
all  four  context  scales;  the  five  outside  searchers  completed 
the  public  knowledge  scale  and  the  internal  knowledge  scale; 
and  the  project  searcher  completed  all  four  scales i  the 
first  project  search  (first  digit  of  the  code,  15  was  the 
only  project  search  to  examine  the  effects  of  context  oh  the 
search. 
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IMSF  PROJECT  iST-S50  5411 
EXPERIMENTS  GIM  THE  COSNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  001 

DIALOG  DATABASE  USED:    11  -  PSYCH1NF0 

SEARCHER  CODE  NUMBERS:  013*  005,  002,  016,  021,  119,  2l9,  319,  419 
ftl.   BRIEF  TITLE: 

The  structure  and  function  of   interpersonal   rel ationships i 


A2.   QUESTION  STATEMENT: 
Structure  - 

(4hat  salient  variables  comprise  the  relationship  between  middle  aged  children 

and  their  parents? 

Fuhctioh— 

l^hat  are  tthe  communication  processes  which  are  enacted  in  these  relationships 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  broad 

C.  Research  application:   Graduate  study  -  Nursing 

D.  Retrieve  articles  in  English  only  or  any  language:   English*  German 

E.  Years  to  be  searched:   No  limits 

F.  DIALOG  databases  suggested:    psychology,   sociology,   education,  gerohtology 
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4.0 


Public 
Knowledge 
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4.0 
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USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  002 

DIALOG  DATABASE  USED:   218  -  NURSING  AND  ALLIED  HEALTH 

SEARCHER  CODE  NUMBERS:      033,   004,   □03,   026,   006^   120,   22CD,   320,  420 


Al.   BRIEF  TITLES 
Organisation  design  variables  in  nursing  department 


A2.   QUESTION  STATEMENT: 
Structures- 

1.  How  are  departments  of  nursing  designed/structured/organized?     (in  th« 
acute  care  setting!* 

2.  Size,   Purpose,   and  Environment 

What  "variables"  <criterias,  parameters)  were  used  to  design  the  department  oi 
nursing? 


3-  How  are  departments  of  nursing  integrated  into  the  general  hospital? 
A.     What  "models"  of  organization  are  used  in  nursing? 


TYPE  OF  SEARCH  REQUESTED 


B-  A  precise  or  a  broad  search:  Broad 

C.  Research  application:     Graduate  study-Nursing ;  dissertation 

Dm  Retrieve  artjjles  in  English  only  or  any  language:  English 

E.  Years  to  be  searched:     No  limits 

F.  DIALOG  databases  suggested: 


CONTEXT  SCALES 


EKLC 
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EXPERIMENTS  ON  THE  COGNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  003 

DIALOG  DATABASE  USED:   64  -  CHILD  ABUSE  &  NEGLECT 

SEARCHER  CODE  NUMBERS:   014,   033,   002,   016,   021,   118,   218,   318,  418 
SI.   BRIEF  TITLE: 

The  use  of  stereotypes  by  health  care  providers  in  the  diagnosis  of  child  abu 


A2.   QUESTION  STATEMENT: 

What  typif icatiohs  of  clients  are  used  by  health  care  providers  in  the 
diagnosis  and   labeling  of  child  abuse? 

tyjDif  icatibnss     characteristics,  stereotypes 
health  care  providers:     nurses  and  physicians 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Broad 

C.  Research  ajspl  ication:     Faculty  research— Nursing 

D.  Retrieve  articles  in  English  only  or  any  language:  English 

E.  Years  to  be  searched:     1975  to  1985 

F.  DIALOG  databases  suggested:     Psych-Info,   Soc.-Info,   Child  Abuse  and  Negled 


CONTEXT  SCALES 
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SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  004 

DIALOG  DATABASE  USED:  154  -  MEDLINE  i9S0-present 

SEARCHER  CODE  NUMBERS:  010,  037,  009,  024,  028,  120,  220,  320,  420 
Al.   BRIEF  TITLE: 


Con-broiled  lung  hyperinflations,  endotracheal  suctioning,  and  cerebrovascular 
status  in  persons  with  severe  closed  head  injuries 

A2.   QUESTION  STATEMENT: 

Tb^_?i'''  ^^"^he  research  is  to  identify  the  effects  of  control  led  hyperinf  lati^ 
breaths,   delivered  prior  to  and  following  endotracheal   suctioning,   upon  the 
cerebrovascular  status  of  adult  subjects  with  severe  closed  head  injuries. 
Specifically,   two  research  questions  have  been  generated. 

li     What  are  the  effects  of  controlled   lung  hyperinflation  breaths 

prior  to  and  following  endotracheal  suctioning  upon  Mean  Intracrani< 
Pp^^sure   <MiCP),   Mean  Arterial  Blood  Pressure  <MABP!)  ,   Cerebral  Perfusion 
Pressure  and  Heart  Rate  in  adults  with  severe  closed  head  injuries? 

 2.     What  is  the  optimal   lung  volume  delivered  during  control  led  lung 

hyperinflations  that  will   produce  minimal  changes  in  Mean  Intracranial 
Pressure,  Mean  Arterial   Blood  Pressure ,   Cerebral   Perfusion  Pressure  and  Heart 
Rate  in  adults  with  severe  closed  head  injuries? 


TYPE  OF  SEARCH  REQUESTED 

Bi  A  precise  or  a  broad  search:  Broad 

Cm  Research  application:     graduate  study-Nursing 

Dm  Retrieve  articles  in  English  only  or  any  language:  English 

Em  Years  to  be  searched^     No  limits 

F.  DIALOG  databases  suggested: 
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CONTEXT  SCALES 


User 


Problem 
Definition 

A. 5 
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4.0. 
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Knowledge 
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1.0 
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EXPERIMENTS  ON  THE  COSNiTiVE  ASPECTS  OF  iiMFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  DD5 

DIALOG  DATABASE  USED:     148  -  TRADE  AND  INDUSTRY  INDEX 

SEARCHER  CODE  NUMBERS:     dG6,   013,   004,   003,   026,   120,   220,   320,  420 

Al.   BRIEF  TITLE: 

Rules  of  -thumb  in  industry 

A2.  QUESTION  STATEMENT: 

Keywords:     Rules-of -thumb ;   axiom jon-the-average   (other  possiBiiites :  as  a 
general   rule;   expert  opinion.) 

Focus:     I  am  looking   for  rules  of  thumbs   industry  by  industry   (at  the  2-digit 
SIC  level   or  can  be  more  specific!)       These  are  rules  or  axioms  which 
evolved  over  the  yearsw     They  are  generally  known,   not  confidential. 

Example:     In  the  motel /hotel   industry  it  is  known  that  oh  the  average  you 

should  fill  60%  of  your  rooms  or  else  you  lace  financial  problems. 
The  same  rule  or  axiom  applies  roughly  to  airline  seats  and  to  movi^ 
theatres. 

Scope:     All   sectors  include  agriculture;   construction;   mining;  manufacturing; 
all   services   (utilities,   trade,    professional,  repair.*) 


Notes: 

!•     I  am  open  to  use  other  keywords  and  also  to  non-business  data  bases i 

2.  I'd  be  able  to  use  a  search  of   Dissertation  Abstracts  for  theses  which 
focused  on  an  industry   (SIC  2-digit  or  more  likely  narrower.*),   bat  this  may  be 
too  costly  or  cumbersome^ 

3.  A  friend  did  do  a  search  for  me  of  PTS  Prompt  already^  so  i^'d  go  with  Trac 
and  Industry  first.  PAIS  may  yield  something;  also  Canadian  Business,  industi 
Data  Sources,   Management  Contents 

4.  I'd   like  to  know  if  non-business  information  bases  in  Dialog  contain 
business  information. 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Broad 

C.  Research  application:     Faculty  research 

D.  Retrieve  articles  in  English  only  or  any  language:  English 
Eo   Years  to  be  searched:     Last  5  years 

F.  DIALOG  databases  suggested:  Trade  and  Industry,  Industry  Data  Sources, 
Management  Contents 

o 

ERIC 


QUESTION  NUMBER:   005  -  PAGE  2 
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Problem    i=>ubiic  internal 

Definition  intent  knowledge  knowledge 


User 

4.0 

2.0 

4.0 

4.0 

Searcl  ler 

003 

3.0 

3.6 

□OA 

2.0 

4.0 

006 

4.0 

1.0 

013 

4.0 

2.0 

026 

1.6 

1.0 

120 

4.0 

1  .0 

1.6 

1.0 

22$ 


Form  15 


(3/31 /86) 


NSF  PROJECT  lST-i50  S'^^ll 
EXPERIMENTS  ON  THE  COGNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  006 

DIALOG  DATABASE  USEDs     h  -  NTIS 

SEARCHER  CODE  NUMBERS:     Oil,   040,   039,   038,   027,    120,   220,   320,  420 


Al.   BRIEF  TITLE: 

Preventiooon  of  crystal  growth  on  foreign  surfaces 
A2.   QUESTION  STATEMENT: 

Carbon  dioKide  crystals  are  formed  in  a  rea'.jtor.     At  high  production  rate 
solid  growth  on  reactor  interior  surfaces  became  a  major  problem.     We'd  like  1 
find  out  the  cause  of  this  problem  and  various  methods  that  can  be  used  to 
avoid  solid  growth  on  surfaces. 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Broad 

C.  Research  application:  Industrial 

b.  Retrieve  articles  in  English  only  or  any  language: 

Ei  Years  to  be  searched s     No  limits 

F.  DIALOG  databases  suggested: 


Any  language 


CONTEXT  SCALES 
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USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  807 
DiAtde  DATABASE  USED: 
SEARCHER  CODE  NUMBERS: 


75  -  MANAQEMENT  CONTENTS 
026,   QQ6,   01/>,   004,   003,   119,   219,    319,  419 


Al.  BRIEF  TITtE: 

Barriers  to  strategic  haman  resource  management 


A2-   QUESTION  STATEMENT: 

Employee  comrpitment  to  an  organization  develops  as  a  result  of  the  interactic 
between  the  organization's  treatment  of  employees   "Ci.e-   skills,  performance, 
company  policies  regarding  benefits^   career  opportunites) ,   and  the  employees' 
contribution  to  the  company  <i  .e^   ski  1  Is ,   performance ,   abi  1  ities.*)  .     General  1^ 
the  bei ief   is  that  the  greater  the  employees'^  commitment  to  the  organization^ 
the  greater  their  fpotiyation  to  perform  the  task  and  pro jects  required  by  the 
organization;     T?.??^?"^???  employee  commitment ,   and  thereby  maximize 
organizational   perforrpance^  senior  managers  can  no  longer  assume  a  short— tern" 
planning  horizon  toward  employee  issues.     That  is,   they  must  no  longer  wait 
until   a  crisis  arises  to  address  issues  regarding  the  organization's  treatmen 
of  employees Instead^  senior  managers  must  assume  a  long-term,  strategic, 
planning  per^spectiye  that  enables  them  to  not  only  attain  business  objectives 

them  to  enhance  the  development  of  employees  in  the 

organization. 

At  the  present  time^  the  majority  of  U.S.  corporations  react  to  human  resourc 
issues  only  when  they  reach  crisis  levels.  Very  few  companies  plan 
proactiveiy  to  meet  employee  needsj  and  even  fewer  consider  the  need  to  enhan 
employee  development  along  with  attaining  business  objectives.  This  research 
examines  the  barriers  to  managing  employees  and  planning  for  future  human 
resource  issues  in  a  long-term,  strategic  manner.  Specifically,  this  researc 
addresses  the  question: 

What  factors  impede  the  strategic  management  of  human  resources  in  today's 
organizaticns? 

Possible  determinants  include: 

^Status  and  role  of  the  Human  Resource  Function  in  general  9  and  the  senior 
Human  Resource  executive  in  particular. 

'^91?  (P^y  defined  as  the  behaviors  expected  9^ _the  occupant  <s)  of  the  Humai 
Resource  functiort;   this  definition  includes  the  expectations  of  Human 


EKLC 
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Resource's  constituencies  (senior  management,  employees)   regarding  which 
t>^^1iiviors  are  appropriate,   and  the  enactment  or  conduct  of   the  Human  Respurci 
executive.     STATUS  may  b©  defined  as  the  position  in  the  social  system  of  th< 
organization  accorded  the  Human  Resource  staff  and  senior  Human  Resource 
executive^   and  includes  the  characteristics  of  esteem,   power  respect. 
♦Organizational  Norms  regarding  proper  modes  of   interaction  between  managers 
and  employees.     NORMS  may  be  defined  as  the  standards  z»gainst  which  befiavidr 
eyaluatedf  a  shared  view  of  desirable  behavior- 

*Company  Characteristics  including:     its  age,   employee  number,    long— term 
financial   performance,    labor  requirements   (labor  intensity  and  skill  mix), 
degree  of  decentralization,   organization  structure  and  design. 


TYPE  OF  SEARCH  REQUESTED 


Bm  A  precise  or  a  broad  search s  Broad 

C.  Research  application:     Graduate  study-management 

b.  Retrieve  articles  in  English  only  or  any  language:     Any  language 

E.  Years  to  be  searched:     No  limits 

F.  DIALOG  databases  suggested: 
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USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  _ggS 

DIALOG  DATABASE  USED:     154  -  MEDLINE  19S0-present 

SEARCHER  CODE  NUMBERS:     028,   010,   037,   009,   024,   120,   220,   320,  420, 


Al«  BRIEF  TITLE: 

The  effects  of  aerobic  exe.rcise  on  women  who  are  menopausal 


ft2.   QUESTION  STATEMENT: 

The  study  will  address  the  'Effects  of  an  aerobic  interval  training  prdgram  or 
the  health  of  middle-aged  women  who  are  rnenopausal  •     I  will   be  IbbPcing  at 
health  in  terms  of   jDhysiolbgical  ,   psychological,  and  social  parameters. 

Menopausal  women-  women  who  are  middle-aged   (approx,   35-60 !>; who  are 
experiencing  the     cessation  of  menstrual   periods;   and  who  experience  the 
cessation  as  a  natural   process  of  aging. 

Aerobic  interval  training-  physical  exercise  that  is  individually  prescrioed 
for  each  subject,  attention  is  given  to  the  rate,  intensity,  and  duration  of 
the  exercise. 

Physiological  parameter-  heart  rate^  blood  pressure,  maximum  volume  of  oxygen 
cardiac  stress  testing,    ? treadmill  test> 

Social  Parameter-  lifestyle,   health  care  practices 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search s     Not  specified 

C.  Research  application:     Graduate  study-Nursing 

D-  Retrieve  articles  in  English  only  or  any  language:  English 
E-   yff?c*s  to  be  searched:     No  limits 

F.   DIALOG  databases  suggested:     Medline,  Psychinfo 
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NSF  PROJECT  IST-S50  5411 
EXPERIMENTS  ON  THE  COGNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  009 

DIALOG  DATABASE  USEDl     37  -  SOCIOLOGICAL  ABSTRACTS 

SEARCHER  CODE  NUMBERS:  021,  014,  033,  002,  016,  119,  219,  319,  419 
Ai.  BRIEF  TITLE: 

Alternative  human  services  delivery  systems 


A2.   QUESTION  STATEMENT: 

The  classical  model  of  service  delivery  has  been  the  individual  client  and  %t 
individual  casework  in  a  agency  based  office.     The  model   is  best  exemplified 
the  Family  Service  Agencies.     Beginning  about  1960  variations  and  alternative 
to  this  model  have  been  developed.     We  are  hoping  thru  a  series  of 
dempnstratibh  projects  to  design  alternative  models  and  test  them.     What  I 
would  liRe  are  abstracts  describing  alternative  model  which  have  been 
developed  over  the  past  25  years. 

An  example  would  be  the  decentralization  of  the  service  into  the  black  inner 
city  church,   rather  than  offering  service  at  the  agency.     I  cbUld  narrow  the 
tiUestibn  to  model   which  include  the  utilization  of   "natural"  networks  (black 
church)   and  rely  more  on  community  networks,  natural  or  created  by  the  social 
workers. 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Broad 

C.  Research  application:     Faculty  research— appl ied  social  science 

D.  Retrieve  articles  in  English  only  or  any  language:  English 

E.  Years  to  be  searched :  1960-present 

F.  DIALOG  databases  suggested: 


CONTEXT  SCALES 

Problem  Public  Internal 

Definition  Intent  Knowledge  Knowledge 

User  2.0  2.0  4.0  3.0 

Searcher 

002  -  -  2.6  1.0 

014  •-  -  4.0  2.0 

016  -  ~  4. CD  2.0 

021  -  -  3.0  2.0 

dSS  -  -  3.0  i.d 

119  2.6  2.6  2.6  1.0 


Form  IS  (3/31/86) 

NSF  PROJECT  isf -850  5411 
EXPERIMENTS  ON  THE  COGNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  010 

DIALOG  DATABASE  USED:     154  -  MEDLINE  19S0-pre5ent 

SEARCHER  CODE  NUMBERS:  024,  028,  010,  037,  009,  12J,  220,  320,  420 
Al.  BRIEF  TITLE: 

Hot  i  vat,  ion/ oho  ice  in  chemotherapy  decision  making 


A2.   QUESTION  STATEMENT: 

What  motivates  adults  to  choose  to  continue  receiving  chemotherapy  or  to 
choose  not  to  continue  to  receive  cl  emdtherapy? 

Motivation-  hope,   fear,   despair,   depression,   self-esteem,   self  perception,  se! 
actualization,   needs,   goals  drives,   intrinsic  motivation,   extrinsic  mbtivatior 
motivation,   perceived  life  threat,   internal  anxiety*   values,   self  interest. 
Choice-  conflict,   choice  behavior,   perception,   self  (Perception,  subjective 
factors,  decision  making  choice. 

Chemotherapy-  cancer,   neCplasm,   tumor,  drug  therapy. 
Adults-  patients,   clients,   persons,  individuals 

TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Bread 

C.  Research  applicaticn?     Graduate  study-NUrsirig 

D.  Retrieve  articles  in  English  only  or  any  language:  English 

E.  Years  tc  be  searched:     No  limits 

F.  DIALOG  databases  suggested: 


CONTEXT  SCALES 
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NSF  PROJECT  IST-S50  5411 
EXf'ERIMENTS  ON  THE  COGNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  811 

DIALOG  DATABASE  USED:     154  -  MEDLINE  198b-p£ esent, 

SEARCHER  CODE  NUMBERS:     009,   024,   028,   010,   037,   119,   219,   319,  419 


Ali   BRIEF  TITLE: 

Family  response  to  sudden  infant  death  syndrome 


A2.   QUEStibN  STATEMENT: 

Mhat  BVG  "the  *  p5ychO""BniD'tiDna  1  ^nd  puycho—social   responses  of  parents  and 
surviving  siblings  to  thfe  death  of  an  inf ant  due  to  Sudden  Infant  Death 
Syndrome  <SlbS;   also  called  "crib  death")?   and  what  are  the  coping  strategies 
of  these  family  members? 


TYPE  OF  SEARCH  REQUESTED 

Bi  A  precise  or  a  broad  search  5  Broad 

C.  Research  applications     Faculty  research-Nursing 

Dm  Retrieve  articles  in  English  only  or  any  language:     Any  language 

E.  Years  tp  be  searched;  196d-present 

F.  DiALOS  databases  suggested: 


CONTEXT  SCALES 

Problem  Public  IntiErnr.l 

Definition  Intent  Knowledge  Knowledge 
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EXPERIMENTS  ON  THE  COGNITIVE_ASPECTS_OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBERS  012 

DIALOG  DATABASE  USED:     13  -  INSPEC  1977-present 

SEARCHER  CODE  NUMBERS:      OC35 ,    029,   027,   Oil,   040,   119,   219,    319,  Vi9 


Al.   BRIEF  TITLE: 

Chemis-bry  of   Silicon  Nitride  and  Silicon  Carbide 


A2.   QUESTION  STATEMENT: 

I  would  like  to  find  out  what  is  known  about  the  chemical   reactivity  of  Si  lie 

Carbide  and  Silicon  Nitride  ceramic  powders  at  low  temjDeratures  <robm 
temperature) -especial ly  in  aqueous  environments-bbth   liquid  water  and  water 
vapor.     I  want  to  know  what  the  surface  chemistry  of  these  materials  is  <what 
9??^^PS  sre  present  and  how  they  interact  with  the  environment?   and  th 
physical  chemistry  <col loidal   behavior >   of  ultrafine  ceramic  powders  made 
from  these  powders.     Important  studies  wbUld  include  surf ace  titration  (also 
called  titration  of   surface  groups  or  potentibmetric) -   zeta  pbtehtial 
(determination  of  surface  potential),   f locculatibh  ahd/br  sedimentatiph 
behavior,   viscosity   ( viscometry)   ad sorption 5FTIR   CFburier_Transf brm  Infrared 
or  traditional   IR   (infrared)   spectrbscopy ,   XPS/ESCA/AUGER  Spectrbscbpies.  Al 
important  would  be  studies  characterizing  the  reactibh  rate  bf   bulk  Silicbn 
Carbide  and  Silicon  Nitride  with  water-fbr  instance  studies  determining  the 
reaction  rate   (kinetics)  or  decomposition  products  at  room  temperature. 


TYPE  OF  SEARCH  REQUESTED 

Bm   A  precise  or  a  broad  search:  Broad 

C.  Research  application:       Graduate  study-Chemical  engineering 

D.  Retrieve  articles  in  English  only  or  any  language:     Any  language-if  abstra 
in  English 

E.  Years  to  be  searched :     No  limits 

F.  DIALOG  databases  suggested:  Electronics/semiconductor,  Chemistry,  Materia 
(science? ,   Ceramics,   Physical   sciences.  Physics 
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NSF  PROJECT  IST-S50  5^11 
EXPERIMENTS  ON  THE  COGNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  013 

DIALOG  DATABASE  USED:     15  -  ABI/INFORM 

SEARCHER  CODE  NUMBERS:     003,   026,   006,   013,   004,   120,   220,   320,  420 
Al.   BRIEF  TITLE: 

The  effectiveness  of  non-profit  human  service  organization 
A2.   QUESTION  STATEMElvJT: 

How  has  the  concept  "effectiveness"     been  defined  and  measured  in  studies  of 
non-profit  human  service  organizations. 

In  resssearch  on  for-profit  businesses,   it  has  been  defined  in  terms  of  a>  go 
attainment,   b.">  obtaining  necessary  resources  for  survival  arid  growth, 
c?   satisfactory  internal,   balances,   exchanges,   integration,   d)  arid 
satisfaction  of  constituencies.     jvjori— profit  differ  in  nuhlerous  ways  in  these 
aspects  from  from  for-profit  organizations. 

Are  these  approaches  transferable?     Are  there  alternatives? 

TYPE  OF  SEARCH  REQUESTED 

Bi  A  precise  or  a  broad  search:  Broad 

C.   Research  applications       Faculty  research-applied  social  science 
b.   Retrieve  articles  in  English  only  or  any  language:  English 

E.  Years  to  be  searched:     No  limits 

F.  DIALOG  databases  suggested: 
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User 


Problem 
Def  inition 

4,0 


Intent 
2.5 


Public 
knowledge 


internal 
knowledge 

4.0 


Searcher 

003 

004 

006 

613 

Q26 

120 


2.0 


i  .0 


3.0 
3.0 
3.0 
3.0 
3.0 
4.0 


1  ;0 
4  id 
2.0 
4.0 
3.0 
3.0 


ERIC 


Form  15  <3/31/86: 


NSF_PROJECT  IST-S50  5411 
EXPERIMENTS  ON  THE  COGNITIVE  ASPECfsOF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  014 

DIALOG, DATABASE  USED:      151   -  HEALTH  f=>LANNING  AND  ADMINISTRATION 
SEARCHER  CODE  NUMBERS:     004,   003,   026,   006,   033,   120,   220,   320,  420 


Al.  BRIEF  TITLE: 

Prospective  payment  system-related  changes  in  hospital  information  systems  ai 
information  systems  groups 

A2.   QUESTION  STAfEMENf : 

The  advent  of  prospBctive  payment  systems  for  American  health  care 
brgahlzatibns  ^including  diagnosis-related  groups  <"bRGs")i>  is  expected  to  he 
caused  many  changes  in  American  hospitals'  Use  of  an  information  systems 
grbup/f unction.  One  expects  the  increased  design  and  development  of  improvec 
rnanagemeht  information  systems  that  will  help  hospital  decisions  makers  manac 
in  the  new  payment  envirdmerit-  These  computer-based  information  systems  wou] 
assist  the  hospital  in  product  and  service  costing  (micro- costing) ,  as  well  c 
better  integration  of  patient  care  information  with  financial  ^charge!> 

information.     One  also  expects  changes  in  the  position,   role  and  reward  

structure  of  the  hospital   inf  ormation  systems  group  -  management  informat: 

systems  become  more  central  to  the  survival  of  the  hospital ,   one  would  expecl 
the  hospital's  data  processing  group  to  become  larger ,   better  Paid,   and  more 
powerful-     They  are  also  expected  to  be  more  closely  linked  with  top  hospital 
decision  makers- 
Terms: 

Prospective  paymefit/DRG:     systems  used  by  Medicare,   some  Medicaid  and  other 
third  party  payors  to  set  the  price  for  patient  care  <in  patient)  before 

service  is  rendered  to  the  patient-   

Management  inf brfpation  system/cdmputer-based   information  system/information 
system:     the  cbHectidn  df  hardware,   software  and  procedure  used  to  collect, 

store,   and  report  information  used  in  task  and  decision  support i  

Information  systems  grdup/MIS  group/data  processing :     the  individuals  who 

develop,   acquire,   manage  inf ormation  systems  for  an  organization-  

Micro-oosting :     The  activity  of  examining,   in  detail,   how  many  resources  are 
cdnsumed  in  producing       unit  of  a  product  or  service i  

biagnosis— related-group  (DRG):     a  particular  pi'Dspective  payment  method^  in 
which  a  set  payment  is  made  for  all  oases  of  a  given  disease  or  health  proble 
Amounts  a  hospital  would  be  paid  for  tht»  patient  would  depend  on  what  DRG  the 
patients  case  is  classified  as- 
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QUESTION  NUMBER:  014  -  PAGE  2 


TYPE  OF  SEARCH  REQUESTED 

Bi  A  precise  or  a  broad  search:  Broad 

C.  Research  application:     Graduate  study-Manage 

bi  Retrieve  articles  in  English  only  or  any  language:  English 

E;  Years  to  be  searched:     No  limits 

F.  bi ALOG  databases  suggested: 
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EXPEi=?iMENTS  ON  THE  COSNiTiVE  ASPECTS  OF  INFORMATION 
SEEkiNS  AND  itMFOIRHATiON  RETRIEVING 

LjSEI=?  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:     01 S 

DIAbOS  DATABASE  USED:     154  -  MEDblNE  l98G-present 

SEARCHER  CODE  NUMBERS:     037,   009,   024,   028,   010,   120,   220,   320,  420 


M.  BRIEF  TITLE: 

Retro iental  Fibroplasia 


A2.   QUESTION  STATEMENT: 

Retroiental   fibroplasia  is  a  disease  of  the  eye  which  began  appearing  in 
premature  babies  in  the  mid     to  late  1940"  si     Because  of  excessive  oxygen  ir 
the  incubators^  these  babies  were  permanently  blinded.     The  cause  of  this 
disease  was  isolated  in  the  early  1950' s- 

I  am  interested  in  the  occurrences,   causes,   treatment,   and  prevention  of 
Retrolehtal  Fibroplasia  from  1945  to  1985. 


TYPE  OF  SEARCH  REQUESTED 

B-  A  precise  or  a  broad  search:  Broad 

Cm  Research  application:  General 

Di  Retrieve  articles  in  English  only  or  any  language:  English 

E.  Years  to  be  ©iarched:^  1945-1985 

F.  DIALOG  databases  suggested: 


CONTEXT  SCALES 

Probieffi  Public  Internal 

Definition  Intent  Knowledge  Knowledge 


User  5-0  4.0  4.0  3-0 
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024  -  ^  5-0  5-0 
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037  -  -  5-0  1.0 
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Form  15  <3/31/86) 

NSF  PRGJEet  iST-85Q  5411 
EXPERiMENitS  ON  THE  COGNltiVf'   ASPECTS  OF  iNFOi=?HATlON 
SEEKING  AND  iNFORHATiON  RETRiEViNS 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


©UESTI0N  NUMBER:  Q16 

DIALOG  DATABASE  USED:     11   -  PSYCINFO 

SEARCHER  CODE  NUMBERS:      016,   021,   013,   005,   002,   120,   220,   320,  420 


Al.  BRIEF  TITLE: 

Prediction  of  type  of  activity  during  retirement 


A2i  QUESTION  STATEMENT: 

The  question  here  concerns  1>  what  kinds  of  activities  people  engage  in  dur; 
retirement  and  2!»  what  oharactistics  and  background  factors  prior  to  retiref 
indicate  the  retirement  activity  patterns^  The  notion  here  is  that  eventual 
such  information  would  be  useful  in  pre-retirement  couhselihg. 
i  am  most  interested  in  what  people  do  in  retirement,  although  it  would 
probably  be  relevant  to  consider  how  they  feel  about  what  they  do  but  not  he 
they  feel  about  retirement  in  general . 


TYPE  OF  SEARCH  REQUESTED 

Bi  A  precise  or  a  broad  search:  Broad 

C.  Research  application:       Faculty  research— Psychology 

D.  Retrie've  articles  in  English  only  or  any  language:  English 

E.  Years  to  be  searched:       No  limits 
Fi   DIALOG  databases  suggested: 
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NSF  PROJECT  IST-850  5411 
EXPERIMENTS  ON  THE  COSNlflVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIONS  FOR  0NLir4E  SEARCHES 


QUESTION  NUMBER:  017 

DIALOG  DATABASE  USED:     5  -  BlOSiS  PREVIEWS  1981 -present 

SEAf?CHER  CODE  NUMBERS:     Q40,   039,   038,   027,   Oil,   ll9,   219,   319,  419 


Al.   BRIEF  TITLE: 

Drug  delivery  devices  and  systems 
A2.   QUESTION  STATEMENT: 

I  want  t©  determine  what  types  of   pomps  arid  cbritrol  systems  are  being  used  fc 
delivering  therapeutic  agents  (drags)   in  animal  experiments  and  clinical 
applications. 

Topic  of  intersts  principles  of  specification  of  pomps;  open-loop  vs.  closed 
loop  control  of  these  devices;   clinical   user  experience. 

TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:     Broad-but  orily  less  thari  200  abstracts 

C.  Research  application:     Faculty  research-Biomedical  erigineering 
b.   Retrieve  articles  in  Erigiish  only  or  any  language:     Ariy  language 

E.  Years  to  be  searched ^     Las-c  five  years 

F.  DIALOG  databases  suggested: 
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SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  018 

DIALOG  DATABASE  USED:     15  -  ABI/ INFORM 

SEARCHER  CODE  NUMBERS:      004,   003,   026,   006,   014,   119,   219,   319,  419 


Al .   BRIEF  TITLE: 

Competencies  (skills,   abilities)   needed  in  the  managerial  role 


A2.   -QUESTION  STA'WENT: 

Question:*    What  are  the  key  competencies  that  a  manager  needs  to  perform  well 
in  a  managerial   role,   in  general.    CFrom  these,  we  will   later  apply  these 
general  managerial  competencies  to  the  role  of  the  physician-manager-that  is, 
physicians  who  manage  other  physicians) . 


TYPE  OF  SEARCH  REQUESTED 


Bi  A  precise  or  a  broad  search:  Broad 

C.  Research  application^     Graduate  study-brgani  behavior-  Will   be  uset 
in  dissertation  and  dissertation  results  wiH         used  by  school  of  management 
to  create  t ) a  new  program  for  physician  managers  2)and  for  the  creation  of  a 
model  of  managerial  competencies  to  be  used  with  other  groups  as  well. 

D.  Retrieve  articles  in  English  only  or  any  lancjuage:  English 
Ei   Years  to  be  searched^  No  limits 

F.   DIALOG  databases  suggested:       Maybe-Hanagement  contents 
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NSF  PRGJEiSt  iSt-85Q  5411 
EXPERIMENTS  ON  THE  e06NiTiVE  ASPECTS  OF  iNFORMATlON 
SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  019 

DIAUOS  DATABASE  USED:        75  -  MANAGEMEMT  CONTENTS 

SEARCHER  CODE  NUMBERS:     001,   013,   005,   017,   008,   119,   219,   319,  419 


Al .   BRIEF  TITLE: 

Have  the  IIA  standards  had  an  impact  on  internal  auditing. 


A2.   QUESTION  STATEMENT: 

We  are  interested  in  the  perceived  impact  of  the  standards  for  internal 
auditing  published  by  The  Institute  of   Internal   Auditors  in  1977.     These  are 
voluntary  professional  standards  which  effect  the  scope  of  work,  organisation 
education^   and  independence  of  practicing  internal  auditors.     Any  comparison 
these  internal  audit  standards  to  standards  in  other  business  professions  wou 
also  be  helpful.     Articles^   cases,   bddksi   dissertations,   etc.   that  relate 
problems  and  successes  in  implementing  these  standards  of   internal  audit  in 
specific  companies  would  also  be  helpful. 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search :  Broad 

C.  Research  applicatidh:       Faculty  research-Accounting 

D.  Retrieve  articles  in  English  only  or  any  language:  English 

E.  Years  to  be  searched:  1977-1986 

F.  DIALOG  databases  suggested: 
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EXPERIMENTS  ON  THE  COGNITIVE  ASPECTS  OF  iNFORMATiON 
SEEKING  AND  INFORMATION  RETRiEViNS 


USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  020 

DIALOG  DATABASE  USED:     15  -  ABI/ INFORM 

SEARCHER  CODE  NUMBERS:     032,   005,   001,   017,   013,   119,   219,   319,  419 


Al.   BRIEF  TITLE: 

Financial  statement  presentation 


A2.   QUESTION  STATEMENT: 

(4hat  literature  and  research  exists  regarding  the  presentation  of  financial 
statements  and  their  effect  on  users  of  them?     Of   particular  interest  is 
disclosure  requirement  form  of  the  SEC. 


TYPE  OF  SEARCH  REQUESTED 

B.  A  [srecise  or  a  broad  search:  Broad 

C.  Research  application:       Graduate  study-accounting 

D.  Retrieve  articles  in  English  only  or  any  language:  English 
E-  Years  to  be  searched:     No  limits 

F.   DIALOG  databases  suggested: 


CONTEXT  SCALES 


Problem 
Def  inition 
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Knowledge 
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Knowledge 
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4.0 
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EXPERIMENTS  ON  THE  COSNITIVE  ASPECTS  OF  INFORMATION 
SEEKINS  AND  INFORMATiOM  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  Q2i 

DIALOG  DATABASE  USED:     37  -  SOCiOLOSICAL  ABSTRACTS 

SEARCHER  CODE  NuMBERS:     021,   002,   016,   014,   033,   120,   220,   320,  42Q 


Al i   BRIEF  TITLE: 

Social  supports  of  never  married  and/or  child-free  older  women 


A2.  QUESTION  STATEMENT: 

What  social  sapport  arrangements  and  networks  are  available  to  never  married 
older  women?     What  measares  and  definitions  of  adequacy  of  social  support 
arrangements  have  been  used?     Does  the  physical  and/or  mental  health  of  neve 
married  or  child— free  older  people  differ  from  married  old  parents? 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Broad 

Ci   Research  application:     Faculty  research-Applied  Social  Science 

D.  Retrieve  articles  in  English  only  or  any  language:  English 

E.  Years  to  be  searched:     1965  to  1985 

F.  DIALOG  databases  suggested: 


CONTEXT  SCALES 


Problem  Public  internal 

Definition  Intent  Knowledge  Knowledge 

User  5.0  3.0  2.0  4.0 

Searcher 

002  =  -  4.0  2.0 

014  -  -  4.0  2.0 

016  -  -  2.0  2.0 

021  -  -  5.0  3.0 

033  -  -  3.0  1.0 

120  4.0  4.0  3.0  2.0 
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NSF  PROJECT  IST-jBBO  5411 
EXPERIMENTS  ON  THE  COGNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QOEStibN  NUMBER:  022 

DIALOG  DATABASE  USED:     IQS  -  AEROSPACE  DATABASE 

SEARCHER  CODE  NUMBERS:  Q38,  027,  032,  008,  039,  119,  219,  319,  A19 
Ai .   BRIEF  TITLE: 


Space  commercial ization  market  forecast  to  year  2000 
A2.   QUESTION  STATEMENT: 


The  economic  colonization  of  spacer   sparred  by  the  prospect  of  developing  nev» 
and  high  parity  semicondactors  and  pharmaceaticai s  in  a  Q^^^vity  fr^ee 
environment,  coald  prodace  ^BO  bil  lion  in  commercial   ^eyenties  by  the  year 
20b0i.     The  space  marketplace  is  divided  into  six  major  segments:  __'T^Bterials 
processing ,   sp^ce  communications ,   remote  sensing ,  on-orbit  services ,  groand 
sapport  seyices^  and  commercial   laanch  services,     in  addition  space  markets 
f ^^Y^F^^  constraints  and  incentives  inc lading  space  law,  Congressional 
and  federal   laws,   NASA  policy ,   Strategic  Defense  Initiative,   foreign  space 
competition,   and  financial   climate  factors. 

The  space  shattle  has  helped  pave  the  way i  Experiments  tnaye  saccessfaily  ran 
on  the  Earopean-baiit  spacelabi  The  b.S,  space  station^  to  be  launched  in  th 
199Q^s  ,wi  11  incorporate  a  fif^odif  ied  Sky  lab  f  or  long-term  as©-  _This  permanent 
orbiting  presence  will  mark  the  crossover  point  away  f rom  RSD  dominated  by  a 

companies  to  large-scale  industrial   participation  in  space 
commercial ization. 

Beyond  the  tarn  of  the  centary^  space  indastrialization  will  tarn  to  iarge- 

earth  orbiti     the  P^oj^pts  wii  1  include  t-"5  orbit  space  habitats,  geostationa 
orbiting  solar  Power  satel 1 i tes ,    lanar-based  spa       manufacturing  f aci  i ities^ 
mass  driver  9P  electromagnetic  accelerator  space  transport  systems,   and  space 
fabrication  from  non— terrestrial  materials » 


TYPE  OF  SEARCH  REQUESTED 

B.   A  precise  or  a  broad  search:  Broad 

Cm   Research  application:     Space  conference  P^oQ^^'Ti^Qe'^da 

Di   Retrieve  articles  in  English  only  or  any  language:  English 

Ei   Years  to  be  searched ^     no  1 imits 

F.   DIALOG  databases  suggested: 
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EXPERIMENTS  ON  THE  COGNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  023 

DIALOG  DATABASE  USED:      32  -  MEtADEX 

SEARCHER  CODE  NUMBERS:     015,   034,   035,   041,   030,   119,   219,   319,  419 


Al.   BRIEF  TITLE: 

infiltration  of  sintered  powder  metals  parts 


A2.   QUESTION  STATEMENT: 

Locate  all  references  to  sintered  powder  metals  or  powder  metal  parts 
infiltrated  with  copper  or  bronze. 


TYPE  OF  SEARCH  REQUESTED 

Bi  A  precise  or  a  broad  search^  Broad 

Ci  Research  application:  indastrial 

Di  Retrieve  articles  in  English  only  or  any  language:  English 

E.  Years  to  be  searched:     1970  to  1986 

F.  DIALOG  databases  suggested:     METADEX  -32 


CONTEXT  SCALES 
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SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  024 
DIALOG  DATABASE  USED: 
SEARCHER  CODE  NUMBERS: 


191  - 
007, 


RILA 

012,   023,    025,   036,   120,   220,   320,  A20 


Al.   BRIEF  TITLES 

The  cat  in  sixteenth  century  Italian  art 


A2.   QUESTION  STATEMENT: 

What  is  the  meaning  of  the  cat  as  it  appears  in  paintings  in  Italy  from  1A50 
1600?     When  does  it  first  appear  in  this  context?     Cats  <  usually  one  > 
frequently  appear  in  Italian  Renaissance  paintings  of  the  Nativity  and  of  th« 
Madonna  and  Child.     Why?     Is  the  cat  a  symbol  of  the  Vigin  or  of  the  Christ 
^l^?!^?     is  its  symbolism  positive  or  does  it  have  an  evil  connotation?     Is  t^ 
cat  male  or  female?     Who  is  the  first  to  use  the  cat  in  these  contexts?  Doe? 
the  cat  have  the  same  meaning  when  included  in  paintings  of  saints  or  of  the 
Last  Supper? 


TYPE  OF  SEARCH  REQUESTED 


B.  ^_P5'f9ise  or  a  broad  search:     Not  Specified 

e.  Research  application:     Faculty  research  -  Art 

Dm  Retrieve  articles  in  English  only  or  any  language: 

E.  Years  to  be  searched:     No  limits 

F.  DIAL06  databases  suggested: 


Any  language 
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NSF  PROJECT  IST-~850  5411   

EXPERIMENTS  ON  THE  COGNITIVE  ASPECTS  OF . INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  025   

DIAtOS  DATABASE  USED:     1  -  ERIC_  _ 

SEARCHER  CODE  NUMBERS:  002,  033,  016,  021,  014,  119,  219,  319,  419 
Al.   BRIEF  TITLE: 

Relationship  of  oral   and  written  communication 


A2.   QUESTION  STATEMENT: 

What  relationship  is  there  between  the  oral  and  written  language  of  basic 
writerss  (composition  students) ? 

What  evidence  is  there  that  speech   Coral  cbmmunicatidn)   instruction  combined 
with  writing   (composition!)   instruction  enhances  the  quality  of   student  writir 

Possible  search  words: _  Writing,  SpeaRing ,   Composition ,   College  Composition, 
Freshman  Composition,   Basic  Writers,   Oral   Communication,   Speech  Communicatior 
Written  Communication 


TYPE  OF  SEARCH  REQUESTED 

B«  A  precise  or  a  broad  search:  Broad 

C.  Research  ajDpl icatiph :     Graduate,  study  -  English 

D.  Retrieve  articles  in  English  only  or  any  language:  Any  language 
E-  Years  to  be  searched:     No  limits 

F.  DIALOG  databases  suggested: 
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EXPERIM£NTS_ON  THE  COGNITIVE  ASPECTS  OF  INFOKK  ION 
SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIOhS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  02S 

DIALOG. DATABASE  USED:     3B  -  AMERICA :   HISTORY _&  LIFE 

SEARCHER  CODE  NUMBERS:     012,   023,   025,   036,   015,   119,   219,   319,  A19 


Al.   BRIEF  TITLE: 

Workers'   compensation  in  Ohio  and  Ontario 


A2.   QUESTION  STATEMENT: 

What  was  the  interaction  between  the  "law"  and   "lawyers'*     on  the  one  hand  and 
the  policy  of  creating  an  administrative  agency  for  purposes  of  compensating 
industrial   workers  in^yired_br  killed   in  accidents  in  Ohio  and  Ontario  in  the 
years  from  1915  to  1935?     Both  Ohicp  and  Ontario  created  workers'  compensation 
schemes  that  were  effectiye  by  1915-     In  Ontario  the  Workers  Cornpensation  Boar 
and  in  Ohio  the  Industrial   Commission  were  intended  to  operate  as  an 
alternative  to  judicial  determination  pf   liability  in  tort   '^comrpon  i aw) 

litigation  —  the  previous  existing  method  of  compensation^  Ontario 

successful ly  ridded  its  process  of   lawyers  and  courts:   personal   injury  cases 
were  determined  on  a  no-fauit  basis  from  a  compulsory  compensation  fund 
gathered  by  assessments  on  industries-     Ohio  legislated  a  similar  scheme,  but 
allowed  appeals  from  the  industrial   Commission  findings  and   left  room  for 
lawyers  to  represent  clients  before  the  Commission .     I  am  interested  in 
legislative  debates  concerning  this  topic^   in  laws  enacted,   in  laws  applied  by 
courts,   in  bar  associations  or  other  lawyers'   groups  impressibnE:-  of  changing 

circumstances^   in  medical  concern  about  examinations  of   injured  and  dead  

workmen*   in  manufacturers'   associations  and   labor  unions'    ideas  and  reactions, 
in  reports  and  other  materials  produced  by  the  compensation  a  ; 3ncies ,   and  in 
social  scientists  and   legal   scholars'   comments  on  the  issues. 

TYPE  OF  SEAftCH  REQUESTED 

Bm  A  precise  or  a  broad  search:  Broad 

€.  Research  application:     Faculty  research  -  History 

D.  Retrieve  articles  in  English  only  or  any   language:  English 

E.  Years  to  be  searched:     No  limits 

F.  DIALOG  databases  suggested: 
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EXPERIMENTS  ON  THE  COGNITIVE,  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  027 
DIALOiS  DATABASE  USED: 
SEARCHER   CODE  NUMBERS: 


13  -  INSPEC  1977-present 
027,    Oil,    OAO,    005,    025,  120, 


1^0,   320,  426 


Al .   BRIEF  TITLE: 

^igh  pressure  transducer   Cor  sensors.'J 


A2.   QUESTION  STATEMENT: 


Want  to  know  what  principles  hskve  been  used  for  high  pressure  Cgreater  than 
2000  psi)  sensors.  What  results  obtained?  What  are  the  major  problems  for 
designing  a  Miniature  High  Pressure  Sensor? 

TYPE  OF  SEARCH  REQUESTED 

^_P???i^e  or  a  broad  search:  Broad 
^*    f'?^?^?*^^  ^PP^^^^^^^^v    Faculty  -  Electrical   Engineering  and  Applied  Physics 
b.    Retrieve  articles  in  English  only  or  any  language:     Any  language 
E.   Years  to  be  searched No  i i mi ts 
Fm   DIALOG  databases  suggested:  ' 
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NSF  PROJECT  IST^asd  5411 
EXPERIMENTS  ON  THE  COSNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 

USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBERS  028 

DIALOG  DATABASE  USED:      38  -  AMERICA:   HISTORY   &  LIFE 

SEARCHER  CODE  NUMBERS:      036,   007,    012,    023,    034,    119,   219,    319,  419 
Al .   BRIEF  TITLE: 

History  of  University  Circle  in  Cleveland 
A2.   QUESTION  STATEMENT': 

I  am  Studying  the  history  of '  University  Circle  in  Cleveland^   1800  -  1985-  I 
?9*^y^?D9_°D  three  themes:   philanthropy,   city  planning,   arid  public  vs.  private 
development  of  the  land  and  institutions.     To  highlight  these  themes  the 
resulting  book  will  consider  periods  of  conflict  regarding  each  of  these 
themes^ 

TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Broad 

C.  Research  application:     Faculty  research  -Interdisciplinary 

D.  Retrieve  articles  in  English  only  or  any   language:     Any  language 

E.  Years  to  be  searched:     No  limits 

F.  DIALOG  databases  suggested: 
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SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER :  029 

DIALOG  DATABASE  USED:     8  -  COMPENDEX 

SEARCHER  CODE  NUMBERS:     040,   039,   038,   027,   Oil,   120,   220,   320,  420 


Al .  BRIEF  TITLE: 

Microwave  firing  of  ceramics 


A2.   QUESTION  STATEMENT: 

MICROWAVE  APPLICATIONS  TO  CERAMIC  FiRiNG/SINTERiNG 

The  question  is  concerned  with  the  microwavo  firing  or  sintering  of  the 
following  ceramic  materials:   alpha-alum:ina  or  just  plain  alumina^  bariam 
ferrite,   silicon  nitride,   boron  nitride,   titanium  nitride,   titanium  carbide, 
silicon  carbide,   titanium  boride,   and  molybedum  silicide. 

Because  these  compounds  are  sO  specific  the  question  h£.s  been  generalised  to: 
Has  anyone  reported  data  on  the  firing  or  sintering  of  Cary)  ceramic  material 
using  microwave  radiation? 

i  am  unsure  of  the  correct  key  words  for  the  above  search  h^vfc  already 

found  several   relevant  articles  in  t^?_  ^  ■  _  _  ^         sfe^arching  for 

additional   material   specif icaliy  on  the  microwave  properties;  of  the  j^eramic 
materials.     The  articles  I  have  already  found  are  listed  b^lo^/^j         itwi  r^ope  th 
th©y  might  help  qualify  the  key  words  one  should  be  using  to       ?roh  f .  r  slmil 
articles  or  reports. 

Definitions: 

firing  -  to  heat  a  ceramic  gel   so  that  it  hardens  into  ^  hard  ,v  Btfe^i. .  ^al 

sintering  -  to  mix  a  ceramic  with  another  material  .     The  resulti:.9    uxx  is 
heated  until  the  composite  material  hardens. 

microwave  radiation  -  any  electromagnetic  radiation  between  5C30  MHz  and   10  GH 


ERIC 


QUESTION  NUMBER:  029  -  PAGE  2 


REFERENCE  ARTICLES  ALREADY  FOUND 

D.  Johnson  and  R.  Rizzb,  "Plasma  Sintering  of  Beta-Alumina**,  Ceramic  Bulletir 
Vol.   59,   No.   4,   1980,    p.  467-472. 

E.  kemer  and  D.   Johnson,    ''Microwave  Plasma  Sintering  of  Alumina",  Ceramic 
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Mi  Krage,  ''Microwave  Sintering  of  Ferrites",  Ceramic  Bulletin,  Vol.  60,  No.  1 
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Resonator  Applicator",   J.   Annerican  Ceramic  Society,   Vol.   66,   No.   12,    p.  855- 
859. 


R.  Roy,  S.  Komarneni,  and  L.  Yang,  **Controlled  Microwave  Heating  and  Melting 
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QUESTION  NUMBER:  _D30 
DIALOG  DATABASE  USED  5 
SEARCHER  CODE  NUMBERS: 


71    -  MLA  BIBLIOGRAPHY 
025,    036,    007,   012,    023,    120,   220,    320,  420 


Al.  BRIEF  TITLE: 

Creative  evasion  of  censorship  in  South  Africa 


A2.   QUESTION  STATEMENT: 

By  what  devices  and  techniques  do  South  African  artists^   writers^  journalists 
and  academics^   editors  and   publ ishers  seek  to  maximize  freedcsm  of  political 
expression   (either  within  or  beyond  the   law.')?     Presently  in  South  Africa  there 
is  a  great  deal  of  ferment  over  the  matter  of  censbrshijD  of   political  materia 
and  artistic  expression  with   pol itical  content .     Materials  once  banned  are 
being  unbanned.     Matter  which  in  recent  earlier  times  would  have  tjeen  swift^ly 
muffled,     manages  to  survive.     How?     Why?     In  this  dynamic  context  there  is  1. 
tendency  toward  hit  and  miss  publication  and  prosecution.     It  is  a  situation 
that  encourages  creative  evasion. 

There  is  a  great  deal   published  on  South  Africa's  press,   but  the  focus  is  on 
state  control   and  censorship.      It  concentrates  on  legal   issues.     I   prefer  tb 
examine  the   less  formal   practices  of   testing  the  realm  of   the  acceptable,  and 
perhaps,   the  hon-prosecutabl e .     Thus  I  shall    look  at  sel f -censorship , 
"surrogate  censorship",    professional   self-regulation,   and  creative  evasions  oi 
state  control . 

For  case  study  data  I  shall  focus  on  a  number  of  specific  issues  -  The  banninc 
and  unbanning  of  a  novel,   a  newspaper  editor  who  prints,   probably  illegal lyj 

remarks  of  a  banned  person,   a  playwright  who  uses  satire  to  criticize  the 
regime,   etc^     Most  of  my  data  wi 1 1  come  from  interviews  and  an  examination  of 
primary  materials.     But  I  do  need  some  theoretical   background.     What,  if 
anything,   has  been  said  about  these  sorts  of   questions  in  other  English- 
speaking  countries? 
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TYPE  OF  SEARCH  REQUESTED 

B-   ^.P^^'^ise  or  a  broad  search:  Broad 

€•   Research  application:     Faculty  research  -  Political  Science 
b.  Retrieve  f^^i^^fs  in  English  only  or  any  language:  English 
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QUESTION  NUMBER;  031 
DIALOG  DATABASE  USED: 
SEARCHER  CODE  NUMBERS: 


61   -  LISA 

023,    025,    036,    007,    012,   120,    220,   320,  420 


Al.   BRIEF  TITLE: 

Budgeting  for  law  libraries 


A2.   QUESTION  STATEMENT: 

What  are  the  Principles  for  determining  budgets?     What  role  should  adquisitior 
librarians  pi^V'^     ^*^?'^_^'^?'^??'^?9^_?r®_"^?^9^   in  compii ing/pro jecting  budgets? 
How  should  acquisitions  departments  monitor  budgets?     Should/how  should  one 
automate  an  acquisitions  budget? 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Broad 

C.  Research  application:  Publication 

D.  Retrieve  articles  in  English  only  or  any   language:     Any  language 

E.  Years  to  be  searched:     No  limits 

F.  DIAL06  databases  suggested: 
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QUESTION  NUMBER:  032 

DIALOG  DATABASE  USED:      8  -  COMPENDEX 

SEARCHER  CODE  NUMBERS:      029,   032,   001,    008,    017,   119,   219,    319,  419 


Al.   BRIEF  TITLE: 

Fly  ash  as  a  construction  material   in  civil  engineering 


A2.   QUESTION  STATEMENT: 

The  main  purpose  of  this  search  will   be  to  obtain  a  comprehensive   listing  af 
literature  on  the  engineering  properties  and  various  utilizations  of   fly  ash 
and  conditioned  fly  ash  using  cement  or   limei     Major  uses  for  fly  ash  and  fly 
ash  admixtures  that  are  of   particular  interest  include:   i)    load  —bearing 
fills^   2>   structural   backfills,   3)   soil   modifiers,   and  A)   grouting  fillers. 
The  literature  review  will   focus  on  the  following  items: 

1)   factors  affecting  utilization 
2!>   factors  affecting  selection 
3.*>   design  criteria 
4!)   testisig  procedures 
S:>   evaluation  technique 
6.*»   construction  procedure 
75   quality  controls 
8.)   case  histories 

In  summary,   the  search  should  document  research  results  and  construction 
experience  of    lime  and  cement  stabilized   fly  ash,   fly  ash-soil   mixture^  and 
natural   fly  ash.      It  will   provide  a  corriprehensive  review  of  physiochemical 
properties  of   fly  ash  and   its  various  conditioned  admixtures.  Major 
engineering  properties  including  the  compaction  characteristics,    stress  - 
strain  —  strength  -time  relationship,   compressibility,    permeability,  capillar 
action,    frost  susceptibility,   erodibility,   and   leaching  will   also  be  covered  i 
the  search. 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Broad 

C.  Research  application:  Faculty  research  -  Civil  Engineering 
p.  Retrieve  articles  in  English  only  or  any  language:  English 
E.   Years  to  be  searched:     No  limits 

F^^.   DIALOG  databases  suggested: 
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USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER :  _033 

DIALpG_pATABASE  USED:      &  -  COMPENDEX 

SEARCHER  CODE  NUMBERS:  008,  001,  017,  029,  032,  119,  219,  319,  419 
Ai.   BRIEF  TITLE: 


Slurry-flow  friction  factots  derived  from  volume-average?d'  equations 
A2-   QUESTION  STATEMENT: 

The  form  of  friction-factor   (drag  coefficient^  cc^- relations  for  2-ph«<se  s.lurr 
flow  in  pipelines  is  based  on  analogy  with  pipeii..^?  flow  of  single-phase 
f luids. 

Recently   <last  15  years)   volume-averaged  governing  equations  have  been  develo 

for  general   multiphase  systems.     It  is  bur  intention  to  use  these  vblurrie- 
averaged  equations  as  a  b^si     for  determining  the  functional   form* that  slurry 
flow  correlations  should  have- 

The  question  that  we  need  to  know  the  answer  to  is  "Has  this  been  done  before 
Key  words: 

Volume-averaging 
Multiphase  Theory 
Two-phase  flow 
Slurry  and  bubbly  flows 
Friction  factors 
Drag  cpef  f  iciehts 
pimensibna 1  analysis 
Sett  1  irig 

Pneumatic  conveying 

TYPE  OF  SEARCH  REQUESTED 

B,  A  precise  or  a  broad  search:  Broad 

C.  .  Research  application:     Graduate  study  -  Chemical   Engineering    (Proposals  - 
NSF-  DOE> 

p.   Retrieve  articles  in  English  only  or  any  language:  English 

E.  Years  to  be  si^arched:     1976  to  1986 

F.  DIALOG  databases  suggested: 
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QUESTION  NUMBEiR:  034 

DiAhOG   DATABASE  USED:      13  -  INSPEG  i977-present 

SEARCHER   CODE  NUMBERS:      017,    008,    029,    032,    001,    119,   219,    319,  419 


Al -   BRIEF  TITLE: 

A  type  of  expert  systr 

A2.   QUESTION  STATEMENTS 

Are  there  any  rule  based  expert  systems  in  which  the  inference  engine  does  n 
direct  the  search?  Such  a  system  ^rjAd  .provide  the  user  with  information  t 
direct  the  searcn  himself. 

An  expert  system  is  a  computer  program  tSr. t  gives  advice. 
Rule  based  systems  are  based  on  d  large  L^c  of   logical  rules. 

Ah  inference  engine  is  the  part  of  the  program  that  computes  logical  inferen 
from  facts  entered  by  the  user  and  from  other  inferences. 

Inference  engines  commonly  direct  the  search  by  a  method  described  as  backwa 
chaining. 

TYPE  OF  SEARCH  REQUESTED 


B.  A  precise  or  a  broad  search:  Precise 

Ci  Research  application:      Faculty  research  -  Electrical  Engineering 

Dm  Retrieve  articles  in  English  only  or  any   language:     Any  language 

E.  Years  to  be  searched:     No  limits 

F.  DIALOG  databases  suggested: 
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QUESTION  NUMBER^  _035 

DIALOG   DATABASE  USEDj     154  -  MEDLINE  1980-present 

SEARCHER  CODE  NUMBERS:     OOS ,   017,   001,   029,    032,   120,   220,   320,  A20 


Al.   BRIEF  TITLE: 

The  therapeutic  effect  of  music  on  oncology  patients 


A2.   QUESTION  STATEMENT: 

Music  therajDy  is  the  use  of  music  to  positively  influence  behaviors     it  has 
been  used  tos   reduce  blood  pressure ^    lower  heart  rate,   decrease  pain  or  act  c 
a  distractdr  of   pain,   increase  verbal  expression  and  participation,  boost 

^nd  most  recently  act  as  a  means  of   preparation  for  visualization  anc 
imagery  techniques.     What  ars  the  physiological   and  emotional  effects  of  musi 
on  the  patient  hospitalized  with  a  chronic  illness?     How  is  music  therajSy 
currently  being  used  171  such  settiiigs?     How  is  the  progress  or  effect  of  the 
music  therapy  sessions  being  measured? 

For  the  passive,  comatose,  or  physically  limited  patient  music  has  subliminal 
effects.  How  can  thr.s  be  empioyed  to  benefit  the  psychological  well-being  oi 
patient? 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise   vr  a  broad  search:  Broad 

e.  Research  application:     Graduate  study  -  Music 

D.  Retrieve  ar.x^ief  in  English  only  or  any   language:  English 

E.  Years  to  be  searched |     No  limits 
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QUESTION  NUMBER:  036 

DIALOG  DATABASE  USED:      90  -  FOREIGN  TRADE  &  ECON  ABSTRACTS 
SEARCHER  CODE  NUMBERS:      012,    023,    025,   036,    007,   120,    220,    320,  420 


M.  Bi=?iEF  TITLE: 

Industrial   policy  in  UJestern  Europe 


A2.  QUESTION  STATEMENT: 

I  am  in'bereBted  in  all   sources  relating  to  the  above  title,   but  especially  in 

CI?  Industrial   policy  and  technological  innovation; 

<2)  Industrial   policy  and  re-structuring  of  industries; 

<3!)  Industrial   policy  -  European  economic  community; 

C4)  Industrial  policy  -  Austria; 

c'5:)  Industrial   policy  and  corporatism. 


TYPE  0F=-  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Broad 

C.  Research  application:     Faculty  research  -  Economics 

D.  Retrieve  articles  in  English  only  or  any   language:     Any  language 

E.  Yearsto  be  searched:     Last  5  years 

F.  DIALOG  databases  suggested: 
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QUESTION  NUMBER:  _037 

DIALOG  DATABASE  USED:      16  -   PIS  PROMPT 

SEARCHER  CODE  NUMBERS:      030,   015,    03A ,    035,    OAl ,    119,   219,    319,  A19 


Al .   BRIEF  TITLE: 

Current  awareness 9   OSHA  hazard  c  rrrpliance 


A2.   QUESTION  STATEMENT: 

Training  of  employees   (workers!)   on  the  right  to  know   (RTK)  ^   OSHA  hazard 
compliance  laws,   chemical   safety  in  the  workplace,   handling  hazardous 
material  5 . 


TYPE  OF  SEARCH  REQUESTED 

B-  A  precise  or  abroad  search:  Broad 

e.  Research  application:  industrial 

E).  Retrieve  articles  in  English  only  or  any  language:  English 

E.  Years  to  be  searched:     1983  to  1986 

Fm  DIALOG  databases  suggested:     Federal   Register,   Occupational  Safety  Health 
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QUESTION  NUMBER:  D3S 

DIALOG  DATABASE  USED:     61   -  LISA 

SEARCHER  CODE  NUMBERS:      041,    030,    015,    034,    035,    120,    220,    320,  420 


Al.  BRIEF  TITLE: 

impact  of  new  electronic  technology  in  the  future  of  major  technic,  n.l  reference 
1 ibraries 


A2.   QUESTION  STATEMENT: 

I  serve  as  the  Chairman  of  the  Library  Board  for  the  Engineering  Societies 
library  in  the  United  Engineering  Center  in  New  York   CityiThe   library  is  under 
considerable  at1;ack  by  the  supporting  engineering  societies   ciASME,   AiChE,  IEEE 
ASCE,   AIME:>   for  not  doing  the  exciting ,   glamorous  new  electronic  things  which 
they  feel    libraries  should.    _We  have  a  financial  crisis  on  day  to  day  operatioi 
and  are  restricted  because  of   this.     However,   we  are  planning  to  have  a  capita 
gifts  fund  drive  to  iget  endowment  to  buy  a  computer,   ecjuipment,   and  whatever 
technological   advances  are  deemed  appropriaate .     [4b  need  guidance  on  what  the 
direction  is  in  the  future  of   document  acquisition^   cataloging  ^   storing  ^  and 
information  dissemenation.       This  covers  the  items  like:    Kl)   direct  delivery 
by  optical   scanning  and  digitising  of   sotArce  matei?ial   for  direct  transmission 
over  phone   lines  or  by  satellite^   (2!)   use  of   CD-ROMS  for  information  storage  - 
how  to  transmit  to  user,   who  puts  information  on  the  CD;    (.B)    justification  of 
computer  use  for  card  catalogs  information  storage;  disappearance  of 

traditional   role  of   a  reference   library  this  sise  and  what  will   replace  it 
(:*1  .3  mi  1 1  ion/year  budget,   1.1  million  pages  photocopied/year ,   non-prof  it 
organisation) . 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Broad 

C.  Research  application:  Other 

D.  Retrieve  articles   in  English  only  or  any   language:  English 

E.  Years  to  be  searched:      Last  5  years 

F.  DIALOG  databases  suggested:      ERIC,    CA ,  INSPEC 
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QUESTION  NUMBER:  _D39 

DIALOG  DATABASE  USED:      15  -  ABI/INFORM 

SEARCHER  CODE  NUMBERS:      035 ,    041 ^    030,    015,    034,    119,   219,    319,  419 


Al .   BRIEF  TITLES 


Measurement  of  organizaticnal  environments 


A2-   QUESTION  STATEMENT: 

The  objective  of  my  overall   research  is  to  demonstrate  a  link  between  the 
brganisatibnal   structure  and  organizational  environment  of  corporations.  The 
goal  of  the  DIALOG  search  is  to  uncover  academic. works  which _ identify  salient 
environmental  components  ahd/br  provide  practical  examples  of  how  to  measure 
those  components.     The  relevant  works  will   likely  center  oh  the  organization 
theory  literature  within  the  business  administration  discipline. 

The  insights  gained  from  this  review  of   literature  will   be  used  to  develop  a 
questionnaire  which  will   be  distributed  to  corporate  managers. 

'^'Organizational  environment "  reefers  to  the  external  operating  conditions  of  a 
firmi     These  conditions  include  the  forces  affecting  the  firm  such  as 
competition,   supplier  and  customer  actions,   government  regulations,   and  sociai 
attitudes.     Within  these  broader  ca^^egories  are  thought  to  be  salient 
components  which  are  critical  to  the  firm's  operation.     For  example,   a  saiieni 
component  of  the  competitive  environment  might  include  the  pricing  strategy  oi 
a  competing  firm. 

key  words  include: 

corporation  organization 

f  irfp       -  organizational  structu/re 

company  9^9«^D5  ?^"^?9^r'^^  _?9?"^  

subsid  iary  env  i  ronmenta 1  components 

organization/environment  typology  industry  context 

environmental   forces  industry  envi»^onment 
organi  zat iona 1  envi  ronment 


QUESTION  NUMBER:   039  -  PAGE  2 


TYPE  OF  SEARCH  REQUESTED 

B.  A  precise  or  a  broad  search:  Precise 

C.  Research  application:     Graduate. study  -  Geography 

D.  Retrieve  articles  in  English  only  or  any  language:  English 

E.  Years  to  be  searched:     No  limits 

F.  DIALOG  databases  suggested: 


CONTEXT  SCALES 


Problem  Public   Internal 

bef  ini tion  Intent  Know 1 edge  Know 1 edge 

User  A.O  4.0  4.0  4.0 

Searcher 

015  -  -  4.0  3-P 

030  -  -  0-0  2.0 

034  -  -  4.0  4.0 

035  -  -  4.0  3.0 
041  -  -  4.0  1 .  d 
119                             1.0  4.0  2.0        .  2.C:> 


Form  15  C3/3i/86f> 


NSF  PROJECT  IST-850_5M1_ 
EXPERIMENTS  ON  THE  COGNITIVE  ASPECTS  OF  INFORMATION 
SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER:  _04D 

DIALOG  DATABASE  USED:     16       PTS  PROMPT 

SEARCHE:R  CODE  NUMBERS:     034,   035,    041,   030,   015,    120,   220,   320,  420 


Al.   BRIEF  TITLE: 

technical  development  and  commercial  activity  in  bacterial  cloning  vectors 


A2.   QUESTION  STATEMENT: 

The  purpose  of  this  search  is  to  identify  the  known  or  proposed  techniques  for 
creating  bacterial  cloning  vectors  and  to  identify  present  or  proposed 
commercial  activity  regarding  the  sale,   manufacture,   or  application  of  bacteri 

cloning  vectors. 
Definitions  

Cloning  vector       A  p^^^^  ;  n    i  or  circular  piece  of   DNA   Cdeoxyribonucleic  acid ) 
which  can  be  insertec  bacteria  which  contain  new  genetic  material  <genes> 

which  manufacture  nov-.ii   proteins^     A  cloning  vector  must  contain: 

-  Origin  of  replication 

-  A  strong  promoter 
Selectable  marker 

Origin  of   replication  -  A  location  oh  a  cloning  vector  which  permits  the 
genetic  duplication  of  the  cloning  vector. 

Promoter  -  A  specific  section  of  the  cloning  vector  which  initiates  the 
transcription  of  th©  DNA  into  RNA  and  ultimately  into  proteins .  *'Tr*l^hscription" 
is  the  molecular  equivalent  of  searching  and  reading  a  computer  file  on  a 
magnetic  disk. 

Selectable  marker  -  A  section  of  the  cloning  vector  which  makes  a  novel  protei 
which  can  be  easily  detacted  by  a  researcher  and  which  indicates  that  the 
cloning  vector  is  operational. 

Laboratory  sequence  for  manufacture  of  cloning  vector 
Cultivate  bacteria   CmicrObidlogy ) 

-  Isolate  plasmids  (biochemical  separation!* 

-  Map  piasmids  (molecular  "road  map**) 

-  Develop  transformation   (manipulation  techniques  for  plasmid) 

-  Purify  cell  DNA  (isolate  genetic  material  of  interest) 

-  Purify  eel  1   RNA  polymerase  (transcription) 
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QUESTION  NUMBER:   040  -  PAGE  2 


Restriction  enzymes  -  molecular  ''forceps"  which  can  open  up,  disect,  and 
resection  DNA 

key  terms 

dfene  splicing   (inserting  genetic  material   into  bacteria.'* 
Recombinant  DNA   (manmade  ONA!» 


Key  information  requested 

Scientific  papers  and  review  articles 

-  Patents 

-  Trade  journal  reports 

-  Research  company  annual   reports  or  Dunn' sBroadstreet  reports 

-  Names  of  companirs  or  institutes  associated  with: 

-  genetic  engineering 

-  recombinant  DNA 

-  cloning  vector 

-  restriction  enzyme 

-  biotechnology 


TYPE  OF  SEARCH  REQUESTED 

B.  precise  or  a  broad  search:  Broad   

C.  Research  ajppl icatioh:     Industrial-  Genetic  engineering 

D.  Retrieve  articles  in  English  only  or  any   language:     Any  language 

E.  Years  to  be  searched:     1975  to  present 

F.  DIALOG  databases  suggested:     LOCKHEED,   PREDICAST,  BIONET 


CONTEXT  SCALES 


User 


Problem 
Def  inition 

2.0 


Intent 
5.0 


Public 
Knowledge 

5.0 


Internal 
Knowledge 

4.0 


Searcher 

015 

030 

034 

035 

041 

120 


4«0 


2.0 


5.0 
6.0 
4.0 
3.0 
5.0 
3.0 


5.0 
1  .0 
3.0 
2.0 
3.0 
1.0 


EKLC 


APPENDIX  B 

SUMMARY  OF  SEARCH  RESULTS  FOR  QUESTIONS  QOOl  TO  Q040 
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APPENDIX  B.  SUMMART  OF  SEARCH  RESULTS  FOR  QUESTIONS  QOOl  TO  Q040 


Appendix  B  contains  a  page  by  page  summary  of  the  results 
6btaf.ned  for  each  of  the  40  questions*    Each  page,  summarizing  one 
question i  Includes: 

1*  A  summary  of  items  obtained  for  each  question  vhen  the  search 
results  were  combined  or  merged  for  all  nine  searches 
Including: 

A*  Total  number  of  relevant  Items  retrieved 

B«  Total  number  of  partially  relevant  Iteihs  retrieved 

C.  Total  number  of  relevant  plus  partially  relevant 
Items  retrieved 

D.  Total  number  of  hot  relevant  Items  retrieved 
E«  Total  number  of  evaluated  itpis  retrieved 

F.  Total  number  of  not  evaluated  items  retrieved 
G«  Total  number  of  items  retrieved 

2*  A  summary  of  the  five  utility  measures  assigned  by  the  user 
to  the  question ^ 

3*  Search  results  itemized  by  t      five  outside  and  four  project 
searches*    Searcher  codes,  beginning  with  a  C  are  outside 
searches;  searcher  codes  beginning  with  1»  2»  3»  and  4  are 
project  searches*    Items  A*  through       above  are  used  here 
along  with  6*  Recalls  H*  Precision^  and  I*  through  N*  the 
various  efficiency  measures.. 


Question  No.  UUl 
Database  No.  Oil 


*     Summary  of  Search  Results  * 


# 

Relevant  abstracts  ; 

:  27 

# 

Partially  relevant  ; 

:  46 

Total 

# 

Rel .   or  Parti   reli  ! 

;  73 

# 

Not  relevant  ; 

!  75 

# 

Evalaated  ! 

!  148 

# 

Not  evaluated  : 

!  626 

total 

# 

of   references  ; 

I  774 

Ovprall  precision  : 

.493 

User  evaluations 


User's  time  :   5. DO  hrs. 

Dollar  value  assigned  :  *150.00 

Worth  assigned  :  5 

Problem  resolution  :  4 

Satisfaction  :  5 


iL?archer  eval  uation : 


A? 
B: 
C: 
D: 
Es 
F: 
S: 


Relevant 


H :  Precision 


Partially  relevant  Is   Total  #  commands 


#  Not  relevant  J: 
Tota 1  #  eva 1 uated  K : 

#  Not  evaluated 
Total  #  retrieved  M : 
Recall  N: 


Total  P  cycles 
Total   #  search  terms 
L:  Online  corihept  tima 
Preparation  ti<ne 
Total  time 


SEAR 

A 

B 

C 

D 

E 

F 

6 

H 

i 

J 

K 

L 

H 

N 

002 

3 

2 

1 

6 

9 

15 

0.068 

0.833 

21 

5 

13 

0. 

167 

0.250 

0 

.417 

005 

7 

2 

0 

9 

11 

20 

0.123 

1 . 000 

22 

6 

12 

0. 

370 

0.117 

0 

.487 

013 

5 

3 

0 

8 

28 

36 

0.110 

1 . 000 

11 

3 

13 

0. 

301 

0.333 

0 

.634 

016 

1 

9 

25 

35 

0 

35 

0.137 

□  .286 

14 

1 

9 

0. 

208 

0 . 333 

0 

.541 

021 

11 

30 

41 

82 

443 

525 

0.562 

0.500 

19 

5 

12 

0. 

338 

0.250 

0 

.588 

119 

3 

6 

6 

15 

92 

107 

0.123 

0.600 

9 

4 

9 

0. 

09S 

0.167 

0 

.265 

219 

0 

2 

6 

8 

39 

47 

0.027 

0 . 250 

9 

3 

4 

0. 

065 

0.083 

0 

.148 

319 

0 

1 

2 

3 

41 

44 

0.013 

0 . 333 

8 

2 

12 

0. 

093 

0.083 

0 

.176 

419 

0 

2 

C 

2 

5 

7 

0.027 

1  .000 

14 

4 

6 

0. 

092 

0.083 

0 

.175 

ERIC 
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Question  No.  002 
Database  No.  216 


*     Summary  of  Search  Results  * 


#  Relevant  abstracts  :  37 

#  Partial ly  relevant  :  36 
Total  #  Rel.  or  Part,   rel .  :  73 

#  Not  re 1 evant  :  1 56 

#  Evaluated   :  229 

#  Not  evaluatecd  :  O 
Total  #  of   references  j  229 

Overall  precision  :  .318 


User  evaluation: 


User's  time  :  0^50  hrs. 

Dollar  value  assigned  :  none 

Worth  assigned  :  4 

Problem  resolution  :  3 
Satisfaction  3 


Searcher  evaluation : 


A:  #  Relevant 

B;  #  Partially  relevant 

Cs  #  Not  relevant  

D:  Total  #  evaluated 

E:  #  Not  evaluated 

Fs  Total  #  retrieved 

G:  Recall 


H :  Precision  

i :  Total   #  commands 

J:  Total  #  cycles 

K:  Total  #  search  terms 

L:  Online  connect  time 

M:  Preparation  time 

N:  Total  time 


C 

b 

E 

F 

6 

H 

i 

j 

K 

L 

M 

N 

5 

12 

22 

0 

22 

0.137 

0.455 

13 

4 

8 

0.173 

0.333 

o.5i; 

0 

2 

2 

b 

2 

0.000 

b .  obb 

18 

3 

15 

0.148 

0.100 

0.2? 

4 

7 

25 

d 

25 

0.24  7 

b.72b 

14 

3 

4 

bu2b5 

0.167 

0.37 

8 

61 

85 

b 

85 

0.329 

bi282 

3 

1 

3 

0.066 

0.167 

0.22 

1 

17 

19 

b 

19 

0.027 

0.105 

23 

6 

16 

0.231 

0.133 

0.3£ 

4 

13 

3D 

0 

30 

0.233 

0.567 

12 

2 

2 

b.204 

0.250 

0.45 

21 

32 

69 

b 

69 

0.507 

0.536 

11 

4 

7 

0.127 

0.083 

0.21 

i 

37 

44 

b 

44 

0.095 

b.l59 

17 

5 

12 

0.292 

0.083 

0.37 

21 

32 

69 

b 

69 

0.507 

0.536 

11 

3 

9 

0.173 

0.083 

0.25 
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Question  No.  Dd3 
Database  No.  064 


*     Summary  of  Search  Resu 1 ts  * 


#  Relevant  abstracts  :  36 

#  Partially  relevant  :  47 
Total  #  Rel.   or  Part.   rel.  :  S3 

#  Not  relevant  :  68 

#  Evaluated  :  151 

#  Not  evaluated  :  272 
Total  #  of   references  :  423 

Overall   precision  :  .549 


User  evaluations 


User's  time  :  3.5S  hrs. 

Dollar  value  assigned  s  430.00 

Worth  assigned  :  3 

Prob i em  resol ut ion  :  3 

Satisfaction  :  4 


Searcher  evaluation : 


As  #  Relevant 

B:  #  Partially  relevant 

Cs  #  Not  relevant 

b:  Towal  #  evaluated 

E:  #  Not  evaluated 

F:  Total  retrieved 

Gt  Recall 


H :   P  T'^c  i.  s  i  on 

i :   Tota  i  #  par:  nands 

J :  Tota 1  #  9yc^?s 

K :   Tota 1  #  search  terms 

L :   Online  connect  time 

H :   Preparation  time 

N:   Total  time 


SEAR 

A 

B 

e 

b 

E 

F 

Q 

H 

i 

j 

K 

b 

hi 

N 

002 

9 

is 

26 

53 

id 

63 

d.325 

d.509 

lA 

5 

13 

Oil67 

0 

.250 

OiAl 

01 A 

1 

2 

i 

A 

12 

16 

0.036 

0.750 

A 

1 

6 

d.059 

0 

il67 

0.22 

016 

0 

0 

0 

0 

ll7 

117 

0.000 

d.ooo 

17 

1 

7 

d.536 

0 

i  333 

0.86 

021 

13 

29 

34 

76 

65 

161 

0.506 

0.5S3 

13 

A 

8 

0il65 

0 

.250 

O.Al! 

033 

b 

2 

P 

2 

3 

5 

0.02A 

1  .OOO 

16 

A 

11 

0il67 

d 

.100 

D.26 

lis 

i 

0 

1 

2 

2 

A 

d.dl2 

diSde 

17 

7 

8 

OiAlS 

0 

.  333 

0.75 

218 

12 

8 

15 

35 

SA 

89 

di2Al 

d.B7l 

8 

2 

10 

OilSA 

0 

.  333 

0.51 

318 

7 

3 

i 

ii 

19 

30 

0.120 

0.909 

19 

5 

13 

0 .  A22 

0 

.250 

0.67: 

1 

1 

3 

5 

id 

15 

0.02A 

0.AOO 

2 

1 

3 

O.OAA 

0 

.250 

0.29 

ERIC 


Question  No*  004 
Database  No.  154 


*     St  ramary  al  Search  Results  * 


#  Relevant  abstracts  :  60 

#  Partially  relevant  s  58 
Total  #  Rel.   or  Part-   rel -  :  118 

#  Not  relevant  :  33 

#  ^valunted  :  151 

#  Not  evaluated  :  8 
Total  #  of   references  :  159 

Overall   precision  :  -781 


User  evaluation: 


User's  time    :  4,00  hrsi 

bol lar  value  assigned  :  none 

Worth  assigned  :  5 

Problem  resolution  :  3 

Satisfaction  :  4 


Searcher  evaluation : 


A:  #  Relevant 

B:  #  Partial ly  relevant 

C:  #  Not  relevant  

D:  Total   tt  evaluated 

E:  #  Not  evaluated 

F:  Total   #  retrieved 

6:  Recall 


H:  Precision  

^ '  ^9^^  ^  ^  commands 

J:  Total  #  ?y<^if? 

k  2  ^9^^ 1 _S_search  terms 

L :  '-'i^iirie  connect  time 

M;  Preparation  time 

r!;  Total  time 


SEAR 

A 

B 

C 

b 

E 

!=• 

G 

H 

i 

J 

R 

L 

M 

N 

009 

6 

1 

3 

lb 

i 

11 

0.059 

0.700 

23 

6 

10 

0.145 

0.300 

0.44 

010 

11 

14 

7 

32 

i 

33 

b.212 

0.781 

17 

6 

6 

0 . 303 

0.417 

0.72 

024 

28 

8 

5 

41 

i 

42 

0.305 

0.878 

18 

6 

7 

0.194 

0.167 

0.36 

028 

5 

b 

4 

9 

i 

10 

0 . 042 

0.556 

14 

3 

13 

0.14S 

0.250 

0.39 

037 

12 

12 

lb 

34 

2 

36 

b.2b3 

0 . 706 

5 

5 

0.168 

0.066 

0.23 

120 

5 

d 

b 

S 

b 

5 

0 . 042 

1 .000 

18 

6 

10 

0.350 

0.417 

0.76 

220 

7 

13 

2 

22 

2 

24 

0ii69 

0 . 909 

20 

5 

9 

D.3S8 

g .  250 

0.63 

320 

22 

3b 

13 

65 

1 

66 

0  -.  441 

0.800 

11 

2 

10 

0.342 

0 . 133 

0.47 

420 

4 

i 

0 

5 

b 

5 

0.042 

^  .000 

7 

2 

9 

0.134 

0 . 333 

0.46 

Question  No.  005 
Database  No.  148 

*     Summary  of  Search  Results  * 


Reievant  abstracts 

:  16 

Partially  relevant 

:  23 

Total 

Rel.   or  Part,   rel .  i 

!  39 

Not  relevant  i 

1  48 

Evaluated  I 

:  87 

Not  evaluated  i 

i  0 

Total 

of   references  i 

!  87 

Overall   precision  s 

!  .448 

User  evaluation: 


User's  time 

Dollar  value  assigned 

Worth  assigned 

Problem  resolution 

Satisfaction 


0.67  hrs. 
*1 00 . OO 

4  ■ 


Searcher  evaluation : 


A: 

#  Relevant 

H: 

Precision 

B: 

#  Partia"   v  ievo-it 

I: 

Total  tt  commands 

C: 

5  Not  re/  snt 

J: 

Totel  #  cycles 

D: 

Total   #  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

L:- 

Online  connect  time 

F: 

Total  #  retrieved 

H: 

Preparation  time 

G: 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

M 

N 

003 

7 

15 

11 

33 

0 

33 

0 

.564 

0.667 

10 

4 

4 

0.123 

0 

.167 

0.29 

004 

1 

2 

4 

7 

d 

7 

0 

.076 

0 . 429 

11 

3 

4 

0.202 

0 

iOS3 

0.28 

006 

1 

3 

2 

6 

d 

6 

0 

.103 

0.667 

11 

2 

6 

0.234 

0 

.167 

0.40 

013 

0 

d 

0 

d 

0 

0 

0 

.000 

0.000 

13 

3 

21 

0.345 

d 

.167 

0.51 

026 

1 

0 

3 

4 

0 

4 

d 

.025 

0.250 

•19 

6 

13 

0.446 

d 

.500 

0.94 

120 

6 

9 

7 

22 

d 

22 

0 

.355 

0.682 

14 

4 

6 

0.249 

0 

-.  333 

0i58 

220 

7 

15 

13 

35 

0 

35 

0 

.564 

0.629 

12 

4 

10 

0.285 

0 

.167 

8.45 

320 

9 

4 

7 

20 

d 

20 

0 

.  333 

0.650 

19 

3 

18 

0.228 

e 

.167 

0.39 

420 

0 

6 

27 

33 

0 

33 

0 

.154 

0.182 

7 

3 

3.123 

0 

.083 

0.20 

ERIC 


Qoes-bion  No^  QQh 
Database  Noi  006 

*     Summary  of  Search  Results  * 


#  Relevant  abstracts 

■  11 

#  Partially  relevant 

!  5 

Total 

#  Rel.   or  Part.   rel.  ; 

!  16 

#  Not  relevant  ; 

!  134 

#  Evaluated  ; 

!  150 

#  Not  evaluated  i 

!  88 

Total 

#  of  references  : 

!  238 

Overall   precision  : 

!  .106 

User  evaluation: 


User's  time 

Dol  lar  value  asBigr.Ed 

Woe  bh  assigned 

Problem  resolution 

Satisfaction 


3.00  hrs. 
none 
2 


Searcher  evaluation: 


A:  4»  Relevant 

B:  #  Partially  relevant 

Cs  S  Not*  relevant 

D:  Total  #  evaluated 

E:  #  Not  evaluated 

F:  Total  #  retrieved 

G:  Recall 


Hs  Precision 

I:  Total  #  commands 

J :  Tbta 1  #  eye 1 es 

K :  Tbta 1  #  search  terms 

L :  Onl ine  connect  time 

M:  Preparation  time 

N:  Total  time 


SEAR 

A 

B 

c 

D 

E 

G 

H 

i 

J 

K 

L 

M 

N 

□11 

0 

0 

0 

g 

1 

1 

g.ggg 

0 

.000 

41 

6 

12 

1  .123 

0 .  333 

1 

.9! 

or.? 

0 

1 

67 

6B 

47 

115 

0,062 

0 

.014 

15 

2 

11 

g.lS4 

0.083 

0 

.2< 

C'3B 

g 

0 

2 

2 

1 

3 

0.000 

0 

.ogg 

IS 

4 

13 

0.036 

0.250 

0 

.2i 

P39 

2 

5 

39 

45 

2g 

65 

0-375 

0 

.133 

21 

6 

10 

0.250 

0.250 

0 

.5( 

040 

0 

g 

4 

0 

4 

o.ocn 

0 

.000 

38 

13 

13 

0.670 

0.117 

0 

.72 

120 

11 

0 

1 

12 

1 

13 

0.685 

0 

.917 

4 

2 

6 

0.071 

0.167 

0 

.2: 

220 

0 

0 

6 

6 

3 

9 

0.000 

0 

.000 

12 

3 

6 

0.186 

0.050 

0 

.2: 

320 

0 

0 

6 

6 

3 

9 

0.000 

0 

.000 

8 

2 

10 

0.157 

0.050 

0 

.2( 

A20 

11 

0 

25 

36 

24 

60 

0.688 

0 

.306 

11 

3 

6 

0.189 

0.050 

0 

.2: 

ERIC 


^81 


***************** 

*     Summary  of  Search  Resu  1 
****************************'i^rri/ 


Relevant  abstracts  : 

:  7P 

Partial ly  relevant  : 

:  39 

Total 

Rel .   or  Part,   rel .  ; 

!  109 

Not  relevant  : 

:  40 

Evaluated  : 

!  149 

Not  evaluated  : 

i  335 

Total 

references  s 

:  484 

Overall   precision  : 

:    .  731 

User  evaluatrioni 


User's  time  :  2.00  hrs. 

Eibilar  value  a?   igned  :  none 

Worth  assigned  :  5 

Problem  resolution  :  4 

Satisfaction  :  5 


Searcher  evaluation: 


SEAR 

003 
004 
006 
014 
026 
119 
219 
319 
419 


1 
16. 
O 
1 
3 
14 
22 
19 
41 


5:3 
0 
0 

1 

7 
7 
6 
7 


A: 

#  Relevant 

H: 

P.r'i?cision 

B: 

#  Partially  relevant 

I: 

Total  S  commafidg 

C: 

#  Not  relevant 

J: 

Total  S  cycles 

D: 

Total  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not-  evaluated 

L: 

Online  connect  time 

F: 

TotPi  #  retrieved 

M: 

Preparation  time 

G: 

Recsl 1 

N: 

Total  time 

C         D         E         F  G 

H            i        J  k 

1 

m 

0 

1 
1 

4 

1 

7 
B 


5 
67 
O 
2 
5 
25 
30 
32 
53 


0 

27S 
0 
10 
13 
13 
3 
9 
24 


5  0.036 
345  0.35S 

0  D.OOO 
12  0.009 
IB  0.036 
38  0.193 
33  0.266 
41  0-229 
77  0.440 


O.BOO 
0.582 
0.000 
0.500 
0.800 
0.840 
0.967 
0 . 781 
0.906 


8 
24 

7 

6 
14 

7 
11 
13 

id 


2 
6 
3 
1 
2 
1 
3 


M 


N 


8  0.108  0.25Q  0.35 

14  0.441  0.167  0.6G 

4  0.185  0.250  0.43 

9  0.063  0.117  0.18 

15  0.355  0.500  0.85 

5  0.132  0.2SQ  0.38 
5  0.114  0.25Q  Q.36 

16  0.199  0.25Q  Q.M: 
5  0.105  0.250  0.35 


ERIC 
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Question  No.  008 
Database  No.  154 


*     Summary  of  Search  Results  * 


#  Relevant  abstracts  :  2 

#  Partially  relevant  s  5 
Total  #  Rel.   or  Part,   rel .  s  7 

#  Not  relevant  s  54 

#  Evaluated  :  61 

#  Not  evaluated  2  0 
Total  #  of  references  s  61 

Overall   precision  :  .114 


User  eS'aluation: 


User's  time  g  0.75  hrs. 

Dollar  Value  assigned  :  none 

Worth  assigned  s  3 

Problem  resolution  s  2 

Satisfaction  :  3 


SearcheiT  evaluation : 


A: 

#  Relevant 

'■^ ; 

Fiecision 

B: 

#  Partially  relevant 

J  : 

Total  #  commands 

C: 

#  Not  relevant 

Js 

Total  ^  cycles 

D: 

Total  #  evaluated 

K: 

Total  *r  search!  terms 

E: 

#  Not  evaluated 

'\  « 

0r«lin?3  crjnriect  time 

F: 

Total  #  retrieved 

M: 

Preparatic"/  time 

<i: 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

C 

b 

E 

F 

G 

H 

i 

j 

k 

t 

M 

N 

009 

0 

0 

8 

8 

0 

o 

D.OOO 

o.boo 

12 

? 

8 

0i092 

b.25Q 

0.3A 

□10 

2 

3 

51 

'.a 

0 

56 

0 . 71 4 

0.089 

8 

4 

0.156 

bi233 

b.3S 

CD24 

2 

3 

34 

39 

0 

39 

0.714 

b.lf:3 

9 

2 

4 

0.065 

0iil7 

0.18 

023 

0 

0 

3 

3 

0 

3 

o.obo 

0.000 

5 

1 

4 

bills 

b.25Q 

b.36 

□37 

2 

0 

11 

13 

0 

'13 

0.286 

0.154 

14 

4 

4 

b.iii 

bil33 

0.24 

120 

2 

4 

9 

15 

0 

IS 

0.857 

0.400 

9 

3 

8 

0.222 

0.217 

0.43 

220 

2 

2 

5 

b 

9 

0;571 

0;444 

12 

5 

Id 

0.312 

0.250 

t'.56 

320 

b 

d 

1 

b 

b.bbb 

b.oob 

"r 

S 

12 

bi2'.a 

0.083 

0.34 

420 

2 

41 

b 

0.6S7 

0.146 

7 

2 

11 

O.lHl 

0.333 

y«47 

er|c 


Question  No-  009 
Database  No.  037 


*     Summary  of  Search  Results  * 


V 

Relevant  abstracts  : 

18 

« 

Partially  relevant  : 

Total 

m 

Rel.  or  Part-   rel .  : 

66 

m 

Not  relevant  : 

84 

m 

EvalUatecl 

<50 

m 

Not  evaluated  • 

;  95 

Total 

m 

of  references 

t---'>5 

Overall   precision  4 

.  -'td 

User  e*     1  uat ion : 


User^'s  tiffie  :  3.00  hrs. 

Dollar  value  assigned  :  none 

Worth  assigned  :  4 

Problem  resolution  :  2 

Satisfaction  :  3 


Searcher  evaluation: 


A: 

S  ReleVant 

H: 

Precision 

B: 

S  Partially  relevant 

Is 

Total  #  commands 

C: 

»  Not  relevant 

J: 

Total  #  cycles 

D: 

Total  S  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

L: 

Online  ccnncct  time 

F: 

Total  #  retrieved 

Ms 

Preparation  time 

Gi 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

t 

M 

N 

002 

i 

6 

2 

9 

45 

54 

0.106 

0.778 

24 

6 

18 

0.273 

0.250 

0.52: 

01^ 

2 

29 

37 

97 

134 

0.121 

0.216 

8 

2 

3 

0.111 

0.250 

0.36 

016 

0 

3 

30 

54 

84 

0.045 

0.100 

13 

1 

6 

0.163 

0.500 

0.66; 

021 

1 

3 

**■ 

6 

52 

58 

0.060 

0.667 

13 

4 

8 

0.157 

0.200 

0.35- 

033 

1 

5 

f; 

a 

26 

32 

0 . 090 

1  .000 

9 

3 

11 

0.143 

e.066 

0.20' 

119 

5 

15 

15 

35 

112 

147 

[3.303 

0.571 

20 

5 

8 

0.163 

0^335 

0.49< 

219 

0 

3 

Q 

3 

20 

23 

b.b4B 

1  .000 

5 

2 

3 

0.046 

0.167 

0.2i; 

3 

7 

3 

13 

55 

0.152 

0.769 

6 

2 

4 

0.091 

0.683 

e  =  i7^ 

'♦19 

9 

28 

19 

56 

210 

266 

0.561 

0.661 

11 

4 

7 

0.006 

0.260 

0.25< 

Quest ibvi  No.  010 
Database  No.  15A 

*     Summary  of  Search  Results  ¥ 

♦♦******¥¥♦¥*¥¥¥#¥¥¥¥#*¥**#♦♦♦¥ 


#  Relevant  abstracts  :  4 

#  Partially  relevant  :  15 
Total  #  Rel.   or  Part,   rel .  :  19 

#  Not  re 1 evant  :  1 30 
9  Evaluated  :  149 

#  Not  evaluated  :  310 
Total  #  of  references  :  459 

Overal  1   jDrecision  :  .127 


User  evaluation : 


User's  time  5   5.50  hrs. 

Dollar  value  assigned   :  *5.00 
Worth  assigned  :  4 

P rob 1 em  resb 1 ut ion 
Satisfaction 


Searcher  evaluation: 


A:  #  Relevant  H:  Precision 

B:  #  PartirJly  relevant  I:  Total  #  commands 

C:  9  Not  relevant  J  a  Total  cycles 

P:  Total  #  evaluated  K:  Total   #  search  terms 

E:  i  Not  evaluated  L:  Online  connect  time 

F:  Total   #  retrieved  M:  Preparation  time 

G:  Recall  N:  Total  time 


B 

c 

D 

E 

F 

G 

H 

i 

j 

k 

h 

M 

N 

0 

0 

•1 

3 

4 

0 . 052 

1  .000 

20 

7 

8 

B.lll 

Oil67 

0.27£ 

0 

2S 

2S 

72 

100 

0.000 

0.000 

15 

5 

7 

0.234 

ei200 

0.43^ 

0 

3 

5 

•14 

19 

0.105 

0 . 400 

38 

5 

25 

0.525 

0.167 

0.692 

0 

•18 

.  22 

d.ooo 

0 .  OOD 

11 

3 

9 

0il39 

0.250 

0.389 

1 

2 

5 

•13 

IS 

0.158 

0 . 60D 

1£3 

2 

7 

0il36 

0.200 

0.336 

1^ 

31 

48 

103 

151 

0.895 

0.354 

•16 

5 

lb 

0il21 

0.667 

D.Tsa 

5 

•10 

23 

33 

0.316 

0 . 600 

24 

8 

13 

0i445 

0.167 

D.612 

s 

89 

98 

172 

270 

0.474 

0.091 

17 

3 

12 

0i467 

0.0S3 

0.550 

0 

8 

9 

24 

33 

0.052 

d.lll 

11 

9 

0.215 

0.167 

0.382 

285 


QUeBtridn  No.  Oil 
Database  No-  154 


*     Summary  of  Search  Results  * 


#  Relevant  abstracts  :  9 
«  Partially  relevant  s  25 

Total  «  Rel.   oi?  Part,   rel  .  5  34 

#  IMot  relevant  :  115 

#  Evaluated  s  149 

#  Not  evaluated  z  161 
Total  #  of   references  s  310 

Overall   precision  :  .228 


User  evaluation: 


User's  time 

Dol lar  value  assigned 

Worth  assigned 

Problem  resolution 

Satisfaction 


1 .92 
none 

4 

4 

5 


hr! 


Searcher  evaluation : 


■<r  levant 

H; 

Precision 

3: 

#  Partially  relevant 

It 

Total  #  co:.iiTiands 

C: 

#  Not  relevant 

J: 

Total  S  cycles 

D: 

Total  #  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

t: 

Online  connect  t4.me 

F: 

Total  #  retrieved 

H: 

Preparation  time 

G: 

Recal 1 

N: 

Total  time 

B 

c 

E» 

E 

F 

e 

H 

1 

J 

K 

L 

M 

N 

0 

i 

2 

7 

9 

0 . 029 

Oisee 

5 

1 

14 

0.074 

0.250 

0.32^ 

31 

4b 

58 

98 

0.265 

0.225 

5 

2 

9 

0.170 

0.250 

0.42C 

6 

15 

23 

22 

45 

0.235 

0.348 

9 

1 

4 

0.071 

0.0S3 

0.15^ 

a 

19 

27 

27 

54 

b .  235 

0.296 

12 

3 

lar 

0.263 

0 . 333 

0.596 

1 

1 

2 

3 

5 

0.a29 

0i50e 

9 

3 

8 

0.083 

0.167 

0.25Q 

15 

70 

93 

105 

198 

0.t76 

0.247 

16 

3 

4 

0.2?" 

0 . 333 

0.571 

10 

61 

77 

93 

l7b 

bi4yi 

0.208 

9 

1 

9 

0.166 

0.0S3 

0.249 

1 

id 

12 

9 

21 

0.058 

0^167 

6 

1 

5 

0.052 

0.167 

0 . 219 

1 

14 

17 

13 

3D 

0.088 

0.176 

7 

1 

9 

0.080 

0.2S0 

0.  33(] 

2  m 


Question  No,  D12 
Da-babdse  No.  013 

*     SamfPary  of  Search  Results  * 


#  Reieyant  abstracts  :  _S 

#  Partially  relevant  :  21 
Total  #  Rel.  or  Part,   rel .  :  27 

#  Not  relevant  :  121 

#  Evaluated  :  148 

#  Not  evaluated  :  9C 
Total  #  of   references  :  SiJfc; 

Overall   precision  :  ,m1i'yy2 


User  evaluation: 


User's  time  :   2.50  hrs. 

Do 11 ar  va 1 ue  assigned  :  none 

Worth  assigned  :  3 

Problem  resolution  :  2 

Satisfaction  :  1 


Searcher  eva 1 uat ion  s 


A:  #  Reieyant 

B:  #  Partially  relevant 

C:  4>  Not'  relevant 

D:  Total  #  evaluated 

E:  #  Not  evaluated 

F:  Total  #  retrieved 

G:  Recall 


H:  Precision 

I s  Tbta 1  ^  commands 

J  3  Tbta 1  W  cyb 1 es 

Ks  Tbtal  #  searbh  terms 

L:  Online  cbhhect  time 

M:  Preparation  time 

N:  Total  time 


SEAR 

A 

B 

C 

D 

E 

F 

S 

H 

I 

L 

M 

M 

DOB 

0 

3 

30 

33 

17 

SO 

0.111 

0.090 

17 

0.420 

0.250 

0.6' 

Dll 

4 

9 

31 

44 

38 

82 

0 . 481 

c  V  295 

37 

2 

33 

0.898 

0»25D 

1  .11 

027 

1 

4 

7 

12 

7 

I'i 

1-!.1S5 

0.417 

14 

2 

8 

0.150 

0.333 

Q.4< 

029 

0 

4 

5 

9 

4 

13 

0.148 

0.444 

7 

2 

4 

0.151 

o.5a:=J 

Q.7i 

040 

1 

1 

2 

4 

3 

7 

0.074 

0.500 

41 

6 

61 

0.723 

d.58i. 

1  ,3( 

119 

1 

4 

54 

59 

30 

89 

0.185 

0.084 

11 

3 

16 

0.204 

d .  41  V 

a.6i 

219 

D 

E 

9 

15 

4 

19 

0.222 

0.400 

IS 

3 

32 

0.382 

d.25D 

319 

0 

h 

9 

15 

4 

19 

0 . 22--; 

0 . 400 

14 

2 

31 

0.348 

d.l67 

0.51 

A19 

1 

7 

19 

27 

11 

38 

0.296 

d .  296 

9 

2 

12 

0.202 

0.167 

d.3^ 

mi 

o 

ERIC 


Questrion  No.  013 
Database  No.  015 


******************************* 

*     Summary  of  Search  Results  * 

****************^r************** 


«  Relevant  abstracts     s  7 
#  Partially  relevant     :  36 
Total  #  Rel -   or  Part.   rel.      :  43 


«  Not  relevant  :  106 

.   «  Evaluated  :  149 

#  Not  evaluated  :  14 

Total  #  of   references  :  163 

Overall   precision  :  .288 


User  evaluations 


User's  time    :   3.00  hrs. 

Dol lar  value  assigned  :  *50.bO 

Worth  assigned  :  2 

f=*roblem  resolution  :  2 

Satisfaction  :  2 


Searcher  evaluations 


A:  #  Relevant 

B:  #  Partial ?y  relevant 

C:  #  Not  rsleyant  

D:  Total  #  evaluated 

E:  #  Not  evaluated 

F  s  Tota 1  #  retrieved 

Gs  Recall 


his  Precision 

is  Total  #  commands 

Js  Total  #  cycles 

Ks  Total  ©search  terms 

b:  Online  connect  time 

Mr  Preparation  tifr» 

N:  Total  time 


SEAR 

A 

B 

C 

b 

e 

F 

6 

H 

I 

J 

K 

L 

M 

N 

003 

3 

B 

22 

30 

3 

33 

0.186 

d.267 

29 

6 

18 

0.404 

0.417 

0.82 

004 

1 

•1 

0 

2 

0 

2 

0.046 

1  .  doo 

17 

3 

11 

0.158 

0.150 

o.;]^o: 

006 

2 

6 

19 

29 

i 

30 

0.233 

0.345 

11 

1 

6 

D.145 

0.25Q 

0.39! 

013 

3 

18 

39 

60 

5 

65 

d .  4ss 

0i350 

11 

3 

6 

0 . 206 

0.083 

0.28< 

026 

A 

14 

37 

55 

7 

62 

0.419 

di327 

11 

1 

6 

0.153 

0 . 333 

0 . 48< 

120 

2 

6 

6 

14 

1 

15 

0.186 

0i571 

22 

5 

4 

0.255 

0.250 

0.50! 

220 

2 

b 

2 

4 

b 

4 

d .  d46 

diSdG 

8 

2 

6 

0.154 

0.066 

0.22( 

320 

3 

7 

is 

28 

2 

30 

d  -.  233 

d,357 

20 

5 

23 

0.270 

0.167 

0.43: 

A20 

3 

6 

9 

2b 

2 

22 

0.256 

0.550 

11 

2 

7 

0.154 

0.050 

0.20' 

ERIC 


Question  No.  014 
ba-tabase  No.  151 


*     SLimmary  pf  Search  Results  * 


Relevant  abstracts  i 

i  35 

Partially  relevant  i 

!  71 

Total 

Reli   or  Part,,   rel.  i 

!  106 

Not  relevant  ! 

;  51 

Evaluated  i 

!  157 

Not  evalaated  i 

!       .  2 

Total 

# 

of  references  i 

!  159 

Overall   precision  : 

!  .675 

User  evaluation: 


User' 5  time 

bol lar  value  assigned 

Worth  assigned 

Problem  resolution 

Satisfaction 


0i75  hrs. 
none 
3 


Searcher  eva luation s 


As  #  Relevant    H 

B:  #  Partial ly  relevant  i 

C:  #  Not  relevant   J 

D:  Total  #  evaluated  K 

E:  #  Not  evaluated  L 

F:  Total  #  retrieved  M 

Gs  Recall  N 


Precision 
total  #  commands 
Total  #  cycles 
Total  #  search  terms 
dnl ine  connect  time 
Preparation  time 
Total  time 


SEAR 

A 

B 

C 

b 

E 

F 

6 

H 

i 

K 

L 

H 

N 

003 

16 

26 

9 

51 

i 

52 

0.396 

0.824 

15 

2 

11 

0 . 31 4 

0.583 

0.89" 

00^ 

10 

15 

5 

30 

1 

31 

0.236 

0^833 

18 

2 

11 

0.248 

0 ".  233 

0.48', 

006 

11 

19 

7 

37 

2 

39 

0.283 

diSll 

11 

2 

12 

0.173 

0.167 

0.34( 

026 

0 

1 

32 

33 

0 

33 

0.009 

0i030 

11 

2 

7 

0.158 

0.250 

0.402 

033 

24 

50  . 

14 

88 

1 

89 

0.698 

0.841 

10 

4 

11 

0.138 

0.066 

0.20J 

120 

7 

7 

3 

17 

i 

18 

0.132 

0.824 

10 

3 

6 

0.221 

0.250 

0.47^1 

220 

5 

11 

5 

21 

0 

21 

0.151 

0.762 

10 

3 

16 

0.234 

0.083 

0.31' 

320 

4 

15 

7 

26 

0 

26 

0.179 

0.731 

IS 

5 

19 

0.349 

0.217 

0.562 

420 

4 

4 

1 

9 

1 

10 

0.075 

0.889 

15 

2 

11 

0.196 

0.167 

0.36; 

o 

ERIC 


Question  No.  015 
Database  No.  154 


*     Summary  of  Search  Results  * 


Relevant  abstracts  i 

;  26 

Partially  relevant 

i  66 

Total 

Rel .  or  Part-   rel .  ; 

i  114 

# 

Not  relevant  : 

1  36 

# 

Evaluatecl  ! 

i  150 

# 

Not  evaluated 

:  172 

Total 

# 

of  references  : 

i  322 

Overall   jDrecision  : 

:  .760 

User  evaluation: 


User's  time  :   4.00  hrs. 

Dollar  S'alue  assigned  :  $50.00 

Worth  assigned  s  5 

Problem  resolution  :  4 

Satisfaction  5  4 


Searcher  evaluation: 


A: 

#  Relevant 

H: 

Precision 

B: 

ft  Partially  relevant 

I: 

Total  ft  commands 

C: 

ft  Not  relevant 

J: 

Total  ft  cycles 

0: 

Total  ft  evaluated 

K: 

Total  ft  search  terms 

E: 

ft  Not  evaluated 

L: 

Online  connect  time 

F: 

Total  ft  retrieved 

M: 

Preparation  time 

G: 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

C 

D 

009 

9 

12 

3 

24 

010 

27 

61 

30 

118 

024 

14 

25 

5 

44 

028 

22 

41 

12 

75 

037 

13 

21 

9 

43 

120 

26 

59 

18 

103 

226 

25 

44 

17 

86 

320 

19 

24 

7 

50 

420 

23 

33 

10 

66 

E 

F 

G 

H 

i 

36 

60 

0. 

184 

0 . 675 

13 

144 

262 

0. 

772 

d .  746 

6 

63 

107 

0.342 

0 . 666 

19 

105 

ISO 

0. 

553 

0.640 

7 

52 

95 

0. 

298 

0.791 

9 

lis 

221 

0. 

746 

0 . 625 

16 

112 

198 

0. 

605 

d .  802 

4 

57 

107 

d. 

377 

d.86d 

10 

75 

141 

0. 

491 

0.848 

12 

j  k      t  M  N 

2  S  0il86  Gil67  0.35; 

i  i  0.191   G.133  0.32^ 

i  20  0il78  0.033  0.21* 

1  6  0.227  0.117  0.34^ 

2  3  0i234  Gil33  0.36' 
5  4  0i324  0i333  D-S5' 

1  9  0i252  0.083  0.33! 

2  13  0il96  0.083  0.27* 
2  10  di258  0.100  0.35i 


2dQ 

ERIC 


Question  No.  016 
Database  No.  Oil 


*     Summary  of  Search  Results  * 


Relevant  abstracts     :  2Sf 

Partially  relevant  i 

1  37 

Total 

Rel-   or  Part,   rel .  i 

!  62 

Not  relevant  i 

!  108 

Evaluated  i 

!  170 

Not  evaluated  i 

:  b 

Tota  1 

of   references  i 

1  170 

Overall   precision  ; 

!  .364 

User  evaluation: 


LJser^s  time  :  4.00  hrs. 

Dollar  value  assigned  :  *75-bb 

Worth  assigned  :  4 

Problem  resolution  :  4 

Satisfaction  s  5 


Searcher  evaluation; 


A: 

«  Relevant 

H: 

Precision 

B: 

«  Partially  relevant 

I: 

Total  #  commands 

C: 

g  Not  relevant 

J: 

Tota 1  #  eye 1 es 

D: 

Total  *  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

L: 

Online  connect  time 

F: 

Total  #  retrieved 

M: 

Preparation  time 

G: 

Recal 1 

N: 

Total  time 

B 

C 

D 

E 

F 

G 

H 

21 

3D 

71 

6 

71 

6 . 661 

b.577 

3 

3 

8 

0 

8 

0.080 

b .  625 

4 

37 

45 

0 

45 

0.129 

b .  1 78 

17 

27 

59 

0 

59 

0.516 

0.542 

4 

20 

27 

0 

27 

0.113 

b.259 

3 

_  7 

12 

0 

12 

0.080 

0.417 

7 

11 

24 

0 

24 

□ .  21 0 

b.542 

8 

23 

38 

0 

38 

b .  242 

b .  395 

5 

12 

18 

0 

18 

0.096 

0 . 333 

i 

j 

k 

t 

M 

N 

6 

2 

3 

0 

.058 

0il67 

0.225 

11 

3 

13 

0 

.154 

0ill7 

0i271 

9 

3 

2 

0 

.141 

0ill7 

0.258 

8 

i 

5 

b 

.191 

0i250 

0.441 

7 

3 

4 

0 

.097 

0i25e 

0.347 

IS 

5 

8 

Q 

.333 

0i4l7 

0i7SO 

9 

3 

6 

Q 

.157 

0iO83 

0  i  240 

15 

4 

22 

Q 

.348 

OilOe 

0i448 

11 

2 

12 

0 

.154 

0il67 

0.321 

29i 


Question  No.  017 
Database  No.  005 

*     Summary  of  Search  Results  * 


Relevant  abstracts 

:  36 

Partially  relevant 

:  26 

Total 

Rel .   or  Part,   rel .  ; 

:  62 

# 

Not  relevant  i 

:  88 

# 

Evaluated  i 

:  150 

Not  evaluated  i 

:  377 

Total 

of   references  : 

!  527 

Overall   precision  : 

!  .413 

User  evaluation: 


User' s  time  :  1 .00  hrs. 

Dollar  value  assigned  :  $10.00 

Worth  assigned  :  2 

Problem  resol ution  :  2 

Satisfaction  ;  2 


Searcher  evaluation: 


A;  #  Relevant 

B:  #  Partially  relev 

C:  #  Not  relevant 

D :  Total  #  evaluated 

E:  #  Not  evaluated 

F :  Tota 1  #  retrieved 

6:  Recall 


H:  Precision 

I:  Total  #  commands 

J:  Total  #  cycles 

K :  Tota 1  #  search  terms 

L:  Online  connect  time 

M:  Preparation  time 

N:  Total  time 


SEAR 

A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

Oil 

6 

9 

38 

53 

64 

117 

0 

.242 

0 

.283 

20 

2 

11 

0 

.378 

0 

.250 

0.628 

027 

4 

2 

18 

24 

135 

159 

0 

.096 

0 

.250 

18 

3 

16 

0 

.286 

0 

.167 

0.453 

038 

0 

1 

0 

1 

0 

1 

0 

.016 

1 

.000 

13 

3 

9 

0 

.095 

g 

.217 

0.312 

039 

22 

9 

8 

39 

94 

133 

0 

.500 

0 

.795 

18 

3 

12 

0 

.301 

0 

.250 

0.551 

040 

7 

6 

19 

32 

75 

107 

0 

.210 

0 

.406 

21 

3 

9 

0 

.409 

0 

.117 

0.526 

119 

5 

D 

0 

5 

11 

16 

0 

.080 

1 

.000 

20 

5 

19 

0 

.346 

0 

.250 

0.596 

219 

0 

3 

21 

24 

23 

47 

0 

.048 

0 

.125 

9 

2 

7 

0 

.110 

0 

.167 

0.277 

319 

3 

0 

4 

7 

13 

20 

0 

.048 

0 

.429 

12 

3 

13 

0 

.172 

0 

.083 

0.255 

419 

1 

1 

3 

5 

25 

30 

0 

.032 

0 

.400 

29 

3 

6 

0 

.203 

0 

.083 

0.286 

ERIC 
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Quest ion  No.  018 
Database  No.  015 

«  ^  HK- -N- -N- «  « -N- -N- -N- -N- -N- -N- «■ -t^ 

*     Sammary  of  Search  Results  * 


#  Re 1 e vant  abstracts  :  66 

#  Partially  relevant  :  38 
total  #  Rel.   or  Part,   rel .  :  10^ 

#  Not  relevant  :  46 
S  Evaluated  :  150 

#  Not  evaluated  :  612 
total  #  of   references  :  762 

Overai 1   precision  :  -693 


User  evaiaatibn: 

3.50  hrs. 
none 

3 
3 


User's  time 

E)ollar  value  assigned 

Worth  assigned 

Problem  resolution 

Satisfaction 


Searcher  evaluation: 
'\   


A: 

#  Relevant 

H: 

Precision 

B: 

#  Partially  relevant 

I: 

Total  #  commands 

e: 

#  Not  relevant 

J: 

Total  »  cycles 

D: 

Total  #  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

L: 

Online  connect  time 

F: 

Total  #  retrieved 

M: 

Preparation  time 

G: 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

c 

p 

E 

F 

G 

H 

i 

j 

K 

t 

M  N 

Qd3 

Q 

0 

d 

0 

0 

0 

o.oog 

0.000 

28 

7 

24 

0.438 

0.333  0.77 

dd4 

55 

29 

31 

115 

541 

656 

0.308 

0.730 

29 

5 

26 

0.497 

0.250  0.7^ 

006 

18 

11 

Id 

39 

30 

69 

0.279 

0.744 

15 

2 

6 

0.259 

0.167  0.42 

014 

□ 

d 

d 

0 

D 

0 

0.000 

o.ogo 

5 

2 

5 

d.i93 

0.083  0.27 

□26 

d 

d 

d 

d 

0 

0 

0.000 

O.OOQ 

14 

2 

14 

0.286 

0.250  0.52 

119 

d 

d 

2 

2 

9 

11 

0.000 

0.000 

12 

3 

7 

0.170 

0.500  0.67 

219 

9 

4 

7 

20 

45 

65 

0.125 

0.650 

11 

3 

5 

0.130 

0.167  0.2S 

319 

d 

d 

0 

D 

1 

1 

O.noO 

o.qpq 

9 

2 

13 

0.136 

00.167  0.2 

419 

1 

1 

o 

2 

13 

15 

0.019 

1  .000 

12 

3 

10 

0.205 

0.167  0.37 

o 

ERIC 


Quest ion  No.  019 
Database  No^  (375 


*     Scimmary  of  Search  Results  * 

«***««««««««««««««-if  «««««««««««« 


Total 


Total 


# 

Re 1 evant  abstracts 

:  27 

# 

Partial iy  relevant  ; 

:  49 

# 

Re 1  or  Pa  rt  i   re 1 .  ! 

:  76 

Not  relevant  ; 

1  74 

# 

Evaluated  ; 

1  IBQ 

# 

Not  evaluated  s 

1  B2 

of   references  : 

2Q2 

Overal 1   precision  ; 

.506 

User  evaluation: 


User^s  time      :   1^25  hrs. 

Dollar  value  assigned  :  none 

Worth  assigned  :  5 

Problem  resolution  :  3 

Satisfaction  :  4 


Searcher  evaluation : 


A: 

#  Relevant 

H: 

Precision 

B: 

S  Partially  relevant 

i: 

Total  #  commands 

€: 

#  Not' relevant 

J: 

Total  #  cycles 

b: 

total  #  evaluated 

R: 

Total  #  search  terms 

E: 

#  Not  evaluated 

i_: 

Online  connect  time 

F: 

Total  #  retrieved 

H: 

Preparation  time 

6: 

Recai  i 

N: 

Total  time 

SEAR 

A 

B 

e 

b 

E 

F 

€ 

H 

i 

j 

K 

b 

M 

N 

Q01 

d 

0 

1 

1 

1 

2 

0.000 

d 

iddd 

16 

2 

8 

0.305 

0  i  41 7 

0 . 725 

005 

4 

5 

3 

12 

1 

13 

dills 

d 

.75d 

Id 

4 

9 

0.189 

Oil  33 

d.32S 

008 

6 

15 

15 

36 

4 

40 

di276 

d 

i583 

26 

9 

8 

0i272 

0.133 

d.40£ 

013 

6 

12 

2 

20 

5 

25 

0.237 

d 

i9dd 

8 

1 

5 

0il57 

0il67 

0.32^ 

017 

7 

14 

22 

43 

18 

61 

di276 

d 

488 

11 

3 

7 

0il76 

0.250 

d.42<! 

119 

12 

24 

45 

81 

39 

120 

0i474 

e 

i444 

13 

2 

6 

0i2O9 

0.333 

d.54: 

319 

10 

16 

13 

39 

10 

49 

d .  342 

d 

i667 

11 

2 

9 

0ilO4 

0.066 

dil7t 

219 

12 

4 

3 

19 

2 

21 

0.211 

d 

i842 

14 

2 

6 

0il28 

0.167 

d.29E 

419 

14 

11 

14 

39 

8 

47 

0.329 

0 

.641 

6 

1 

7 

0.072 

0.100 

0.17; 

ERIC 
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Qaestion  No.  Q20 
Database  No.  (315 


*     Sammary  of  Search  Resal ts  * 


Total 


Total 


Relevant  ab5t.rcict.5 

:  26 

# 

Partially  relevant 

:  43 

# 

Rel i  or  Parti   rel . 

:  69 

# 

Not  relevant  i 

!  81 

# 

Evaluated  i 

:  150 

Not  evaluated  i 

!  161 

of   references  ; 

I  311 

Overail  precision  ; 

I  .460 

User  evaluation: 


tJser^  s  time 

bol lar  valae  assigned 

Worth  assigned 

Problem  resolution 

Satisfaction 


iiSQ  hrs. 
none 

5 

3 

5 


Searcher  evaluation: 


A:  #  Relevant 

B:  #  Partial ly  relevant 

€ :  #  Not  re 1 e vant 

E):  Total  8  evaluated 

E:  #  Not  evaluated 

Fz  total  #  retrieved 

6:  Recall 


H: 
i: 


Precision 
Total   #  commands 
J:   Total  #  cycles 
K:   Total  #  search  terms 
b:   Online  connect  time 
M:   Preparation  time 
N:   Total  time 


SEAR 

A 

e 

€ 

E> 

E 

F 

is 

H 

1 

J 

K 

L 

M 

N 

□01 

0 

i 

1 

2 

2 

4 

0iO14 

0.50D 

13 

2 

4 

D.239 

0 . 333 

0.572 

dd5 

1 

0 

3 

4 

5 

9 

0.014 

0.250 

17 

6 

15 

0.346 

0.133 

0.479 

013 

5 

11 

33 

49 

43 

92 

0.232 

0 . 327 

14 

4 

5 

0.141 

0.117 

0.258 

017 

0 

4 

10 

14 

25 

39 

0i05S 

0 .  2S6 

IS 

5 

8 

0.227 

0.167 

0.394 

032 

2 

2 

6 

Id 

8 

is 

0i05S 

0 . 400 

ID 

2 

4 

0.133 

0.083 

0.216 

119 

19 

24 

28 

71 

56 

127 

Qi623 

0.606 

15 

2 

10 

0.252 

0. 333 

0.585 

219 

3 

5 

11 

19 

35 

54 

Oill6 

0 . 421 

21 

4 

12 

0.198 

0.167 

0.365 

319 

i 

2 

2 

5 

13 

IS 

0.043 

0.600 

5 

1 

4 

0.050 

0.066 

0.116 

419 

3 

6 

3 

12 

8 

20 

0il30 

0.750 

16 

4 

9 

0.127 

0.033 

0.160 

ERIC 


g95 


Question  No.  021 
Database  No.  037 

*  Summary  of  Search  Results  * 
***************** 


#  Relevant  abstracts     :  1? 
_       _   #  Partially  relevant     :  _6 
Total  «  Rel.   or  Part.   rel.      :  25 


#  Not  relevant  :  77 

«  Evaluated  :  102 

«  Not  evaluated  :  _□ 

Total  #  of   references  :  102 

Overal 1   precision  :  .245 


User  evaluation: 


User's  tiffie  :  1.25_hrs. 

Dollar  value  assigned  :  *30-00 

Worth  assigned  :  5 

Problem  resolution  :  4 

Satisfaction  :  4 


Searcher  evaluation: 


S:  «  Relevant 

B:  #  Partially  relev 

C:  g  Not  relevant 

D:  Total  #  evaluated 

E:  g  Not  evaluated 

F:  Total  #  retrieved 

G:  Recall 


Hs  Precision 

I:  Total  #  commands 

J:  Total  #  cycles 

K:  Total  #  search  terms 

Ls  Online  connect  time 

M:  Preparation  time 

N:  Total  time 


SEAR 

A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

k 

L 

N 

002 

0 

? 

15 

0 

15 

b .  240 

0.400 

30 

7 

30 

0.387 

0.256 

Gi637 

014 

1 

0 

0 

1 

0 

1 

b.b40 

1  . 000 

11 

3 

4 

0.230 

0.133 

0  i  363 

016 

0 

2 

28 

30 

0 

30 

o.bsb 

0 . 066 

17 

3 

11 

0.416 

0.500 

0.916 

021 

0 

1 

5 

6 

CD 

6 

b.040 

0.167 

21 

5 

is 

0.327 

0.583 

0.910 

C33 

11 

3 

31 

45 

0 

45 

0.560 

0.311 

20 

6 

14 

0.220 

0.066 

0.286 

120 

3 

0 

0 

3 

0 

3 

b.l20 

1  . 000 

13 

3 

12 

0.320 

0  -.  333 

0.653 

220 

4 

0 

1 

5 

0 

5 

b .  1 6b 

0 . 800 

19 

4 

17 

0.443 

0.083 

0.526 

320 

6 

0 

1 

7 

0 

7 

0.240 

0.857 

11 

3 

9 

0.212 

0 . 083 

0.295 

420 

0 

0 

2 

2 

b 

2 

0 . 000 

0 . 000 

11 

3 

11 

0.171 

0.167 

0.338 

296 

ERIC 


Question  No.  022 
ba-babase  No.  108 


*     Sammary  of  Search  Resai ts  * 


Total 


Totai 


# 

Relevant  abstrac'bs 

:  15 

# 

Partially  relevant 

:  135 

Rel .   or  Part,   rel .  : 

!  150 

# 

Not  relevant  : 

:  0 

# 

Evaluated  : 

!  150 

# 

Not  evalaated  : 

!  365 

# 

of  references  ! 

!  515 

Overall   precision  : 

!      1  .0 

User  evaluation: 


Oser'^s  time    :  isOO  hrs. 

Dollar  value  assigned  :  $'id.Od 

Worth  assigned  :  3 

Problem  resolution  :  2 

Satisfaction  :  2 


Searcher  evaluation : 


A:  #  Relevant  H: 

B:  #  Partially  relevant  i: 

€:  #  Not  relevant   J: 

D:  Total  #  evaluated  K: 

E:  #  Not  evaluated  L: 

F :  Tota 1  #  retrieved  M : 

G:  Recall  N: 


Precision 

Tota 1  #  commands 

tota  1  s  9y*^i?^ 

Total   S  search  terms 
Online  connect  time 
Preparation  time 
Total  time 


SEAR 

A 

B 

e 

D 

E 

F 

G 

H 

i 

J 

K 

L 

M 

N 

008 

2 

18 

d 

20 

83 

103 

Oil  33 

1 

iOOO 

14 

4 

22 

0.260 

0.167 

0.42 

027 

6 

60 

0 

66 

68 

134 

0.440 

1 

.  000 

43 

1 

7 

0.213 

0.250 

0.46 

032 

1 

5 

6 

6 

8 

14 

0 . 040 

1 

.000 

9 

2 

8 

0.174 

0 . 083 

0.25 

038 

i 

5 

d 

6 

11 

17 

0i04e 

1 

iOOQ 

18 

6 

Id 

0.148 

0.167 

0.31 

039 

6 

6 

d 

6 

64 

70 

0 040 

1 

iOOO 

26 

6 

9 

0.306 

0.250 

0.55 

119 

12 

113 

d 

125 

154 

279 

0i833 

1 

iOOO 

18 

4 

Id 

0.250 

0. 117 

0.36 

219 

i 

13 

d 

14 

67 

81 

Oid93 

1 

.  000 

11 

3 

15 

0.179 

0.200 

0,37 

319 

d 

2 

d 

2 

0 

2 

0idi3 

1 

iddd 

24 

3 

31 

0 . 34S 

0.250 

D.59 

A19 

2 

13 

0 

15 

97 

112 

0.1 00 

1 

iOOO 

9 

2 

11 

0.113 

0.250 

0. 36 

ERIC 
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Question  Nb»  025 
Database  No.  032 

«««««««««««««« 

*     Summary  of  Search  Results  * 


#  Relevant  abstracts     :  29 

#  Partially  relevant     :  9 
Total  #  Reli   or  Part.   rel.      :  38 


#  Not  relevant  :  51 

#  Evaluated  :  B9 

#  Not  evaluated  :  0 
Total  #  of   references  :  89 

'   Overall   precision  :  -426 


User  cval uation : 


User's  time 

Dollar  value  assigned 

Worth  assigned 

Problem  resolution 

Satisfaction 


1.50 

none 
5 
4 
5 


hrs  I 


Searcher  evaluation: 


A:  #  Relevant 

B:  #  Partial ly  re lev 

6:  #  Not  relevant 

D:  Total  #  evaluated 

Es  #  Not  evaluated 

F:  Total  #  retrieved 

G:  Recall 


H:  Precision 

I :  Total  »  commands 

J:  Total  #  cycles 

K:  Total  #  search  terms 

L:  Online  cphhect  time 

Ms  Preparation  time 

N:  Total  time 


SEAR 

A 

B 

e 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

815 

23 

3 

9 

35 

0 

35 

0.684 

0.743 

22 

3 

6 

6.487 

0.500 

6.98 

036 

1 

1 

4 

6 

0 

6 

0.052 

0 . 333 

12 

4 

4 

.0.147 

0.033 

6.18 

034 

16 

2 

6 

24 

0 

24 

0.474 

0.750 

11 

3 

8 

0.1S3 

0.083 

0.26 

035 

3 

4 

25 

35 

0 

35 

0  =  184 

0.200 

3 

3 

6 

0.117 

0.167 

6.28 

041 

13 

0 

6 

21 

0 

21 

0,342 

0.619 

14 

4 

5 

0.189 

0.250 

6.43 

119 

0 

3 

12 

15 

0 

IS 

0.078 

0.200 

15 

4 

10 

0.172 

0.250 

0.42 

219 

3 

0 

16 

19 

0 

19 

0.078 

0.153 

17 

4 

8 

0.192 

0.066 

6.25 

319 

1 

0 

4 

5 

0 

5 

0.026 

0.200 

7 

2 

5 

0.073 

0.033 

0.10 

419 

18 

2 

7 

27 

0 

27 

0.526 

0.741 

14 

3 

9 

0.103 

0.  083 

6.18 

2B8 
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Question  No.  024 
Database  No.  191 


*    Summary  of  Search  Results  * 


# 

Relevant  abstracts 

:  4 

# 

Partially  relevant 

!  4 

Total 

# 

Rel.   or  Part.   rel.  : 

:  5 

Not  relevant  i 

!  30 

Evaluated                       s  35 

Not  evaluated  i 

!  0 

total 

of  references  ! 

!  35 

Overall   precision  : 

!  .iA2 

User  evaluation: 


User' s  time 

bol lar  value  assigned 

Worth  assigned 

Prob 1 em  reso 1 ut ion 

Satisfaction 


1.25  hrs. 

2 
1 


Searcher  evaluation : 


As  #  Relevant  H:  Precision  

B:  #  Partially  relevant  i:  Total  #  commands 

C:  #  Not 'relevant  J:  Total  #  cycles 

Dg  Total  #  evaluated  K:  Total  #  search  terms 

E  s  #  Not  eva 1 uated  L :  On 1 ine  connect  time 

F:  Total  #  retrieved  M:  Preparation  time 

6:  Recall  N:  Total  time 


SEAR 

A 

B 

C 

b 

E 

F 

<B 

H 

i 

j 

K 

t 

M 

N 

rid? 

i 

2 

7 

id 

D 

10 

0.600 

0 . 30G 

13 

5 

4 

0.289 

0.200 

0 . 489 

012 

1 

1 

A 

6 

d 

6 

d.40Q 

d .  333 

13 

2 

14 

0.267 

0 . 533 

0.600 

023 

d 

1 

10 

11 

d 

11 

d .  2d0 

d .  d9d 

12 

4 

0.334 

Si  250 

0.584 

025 

0 

0 

0 

b 

d 

0 

0.000 

0.000 

8 

3 

5 

0.050 

0il67 

0.217 

036 

1 

2 

8 

11 

d 

11 

d .  6do 

d.273 

14 

3 

17 

0.362 

0ill7 

0.479 

120 

1 

2 

6 

9 

d 

9 

d.6dd 

0.333 

21 

3 

17 

d.2S8 

Q.iP3 

0.438 

220 

1 

i 

12 

14 

d 

14 

b.4do 

Q.i43 

9 

3 

9 

0il61 

Oil;;7 

0.328 

320 

0 

b 

3 

3 

b 

3 

d .  odo 

e.ddo 

5 

2 

15 

0.119 

Oi20O 

0.319 

A20 

1 

2 

16 

19 

b 

19 

d.6bG 

0.158 

40 

5 

4 

0.500 

0iO83 

0.583 

299 
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Question  No.  □25 
Database  No.  001 


*     Summary  of  Search  Resu 1 ts  * 


Relevant  abstraots 

:  30 

# 

Partially  relevant 

:  26 

Total 

Rel .   or  Part"   rel.  s 

:  56 

# 

Not  relevant  ; 

:  94 

# 

Evaluated  : 

!  150 

# 

Not  evaluated  : 

!  430 

Total 

# 

of   references  : 

580 

Overall   preoision  : 

!  .373 

User  evaluation: 


User's  time 
Dollar  value  assigned 
Worth  assigned 
Problem  resolution 
Satisf  action 


2.00  hrs. 
none 

2 


Searcher  eva 1 uation : 


A: 
B: 
C: 
D: 
E: 
F: 
6: 


# 
# 


Relevant 


Hi; 


Partially  relevant  Ii 


#  Not*  relevant  J: 
Total  #  evaluated  K: 

#  Not  evaluated  L: 
Total  #  retrieved  M: 
Recall  N: 


Precision 
Total  #  commands 
Total  #  cycles 
Total  #  search  terms 
Online  connect  time 
Preparation  time 
Total  time 


SEAR 

A 

B 

C 

D 

E 

F 

G 

H 

i 

j 

k 

L 

M 

ivi 

002 

16 

15 

19 

50 

199 

249 

0.554 

0 . 620 

22 

5 

19 

0.359 

0 . 333 

0.692 

014 

3 

2 

6 

11 

48 

59 

0.089 

0.455 

6 

2 

5 

0.094 

0.117 

0.211 

016 

□ 

0 

3 

3 

33 

36 

0.000 

0.000 

17 

2 

12 

0.511 

0.750 

1  .261 

021 

1 

0 

0 

1 

6 

7 

0.017 

1  .000 

12 

3 

8 

0.175 

d .  333 

d.BOS 

033 

0 

0 

0 

0 

1 

1 

0.000 

o.odb 

15 

5 

16 

d.224 

0.083 

0.307 

119 

12 

17 

54 

S3 

112 

195 

0.518 

0.349 

12 

4 

16 

0.254 

0.250 

d.5b4 

219 

O 

O 

1 

1 

18 

19 

0.000 

o.ood 

13 

3 

9 

0.167 

0,167 

0.334 

319 

3 

1 

6 

10 

29 

39 

0.071 

0.400 

10 

2 

13 

0.174 

0.167 

d.341 

419 

1 

0 

14 

15 

35 

50 

0.017 

d .  d66 

9 

2 

7 

0.091 

0.250 

0.341 

ERIC 
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Question  Nd^  026 
Database  No.  038 


*     Summary  of  Search  Results  * 


«  Relevant  abstracts  ! 

!  37 

#  Partially  relevant  ! 

!  39 

Total 

m  Rel.  or  Part,  rel .  ! 

s  76 

#  Not  relevant  ! 

!  3 

*  Evaluated  ! 

!  84 

#  Not  evaluated  I 

1  0 

Total 

S  of  references  I 

i  84 

Overall   precision  I 

I  .904 

User  evaluation: 


User's  time  :  0.83  hrs. 

Dollar  value  assigned  s  none 

Worth  assigned  5  5 

Problem  resolution  :  4 

Satisfaction  s  5 


Searcher  evaluation: 


A:  #  Relevant 

B:  #  Partially  relevant 

C:  #  Not  relevant 

D:  Total  «  evaluated 

E:  #  Not  evaluated 

F:  Total  #  retrieved 

G:  Recall 


hi:  Precision  

I:  Total  #  commands 

J:  Total  #  cycles 

k:  Total  #  search  terms 

t:  Online  connect  time 

M:  Preparation  time 

N:  Total  time 


SEAR 

A 

B 

C 

D 

E 

F 

G 

H 

i 

j 

K 

L 

M 

N 

012 

1 

6 

0 

1 

0 

1 

0.013 

1 .  bob 

9 

3 

5 

0.208 

0.333 

0.54 

015 

0 

0 

3 

3 

0 

3 

b.boo 

b .  boo 

19 

3 

15 

0.255 

0.250 

0.5O 

023 

8 

19 

0 

27 

b 

27 

0.355 

1 .  bbb 

13 

3 

11 

0.218 

0.417 

0.63! 

025 

9 

2 

0 

11 

0 

11 

0.145 

1 .  bbb 

4 

1 

2 

0.036 

0.333 

0 . 36' 

036 

1 

1 

0 

2 

0 

2 

0.026' 

1 .  bob 

8 

2 

9 

0.195 

0.300 

0.49! 

119 

1 

4 

5 

10 

b 

lb 

0,065 

b.sbb 

7 

2 

9 

0.055 

0.250 

0.30! 

219 

29 

25 

0 

54 

0 

54 

0.711 

1 .  bbb 

9 

3 

8 

0.099 

0.083 

0.18: 

319 

1 

0 

0 

1 

b 

1 

0 . 01 3 

1 .  bbb 

11 

2 

27 

0.152 

0.167 

0.31" 

419 

7 

1 

0 

8 

b 

8 

0.105 

1  .bbb 

12 

4 

6 

0.113 

0.200 

o.3i; 

ERIC 


Qaestion  Noi  027 
Database  Noi  013 

»     Summary  of  Search  Results  * 


#  Relevant  abstracts  :  35 

#  Partially  relevant  :  69 
Total  #  Reli   or  Parti   rels  :  104 

#  Not  relevant  :  46 

#  Evaluated   :  150 

#  Not  evaluated  :  58 
Total  #  of   references  :  208 

Overall   precision  :  -693 


User  evaluation: 


User' s  time  

Dollar  value  assigned 
Worth  assigned 
Problem  resolution 
Satisfaction 


12iOG  hrs. 
none 

3 

4 

3 


Searcher  evaluation s 


A: 

#  Relevant 

H: 

Precision 

B: 

#  Partially  relevant 

I : 

Total  #  commands 

Cz 

#  Not  relevant 

J: 

Total  #  cycles 

Dz 

Total  #  evaluated 

K: 

Total  #  search  terms 

Ez 

#  Not  evaluated 

L: 

Online  connect  time 

F: 

total  #  retrieved 

M: 

Prefsaratibn  time 

€: 

Reca 1 i 

N: 

Total  time 

SEAR 

A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

dbs 

7 

11 

1 

19 

12 

31 

Q.173 

0.947 

14 

2 

22 

0.311 

0.167 

b.47< 

Oil 

7 

19 

13 

39 

15 

54 

0.250 

0.SS7 

39 

7 

11 

0.499 

0.250 

0.74* 

027 

8 

12 

11 

31 

7 

38 

0il92 

0.645 

11 

1 

7 

0.157 

0.083 

0.24( 

d2S 

22 

39 

32 

93 

34 

127 

0i587 

0.S56 

11 

3 

_5 

0.133 

0.083 

d.21J 

□40 

19 

14 

2 

35 

12 

47 

0i317 

0.943 

23 

6 

10 

0 . 362 

0 . 066 

0.42i 

120 

2 

0 

0 

2 

1 

3 

0i019 

1  . 000 

4 

2 

3 

o.ibs 

0.083 

0.191 

22b 

6 

7 

7 

20 

4 

24 

0.125 

0.650 

7 

1 

9 

0.162 

0.016 

d.l7« 

320 

22 

38 

32 

92 

34 

126 

0.577 

0.652 

4 

1 

8 

0 . 1 85 

0 . 033 

0 . 21f 

420 

17 

15 

3 

37 

11 

48 

0.327 

0.919 

25 

2 

4 

0.348 

0 . 033 

0.381 

erJc 
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Questibn  No.  028 
Database  No.  038 

*     Summary  of  Search  Results  * 


Total 


Total 


# 

Relevant  abstracts  i 

:  5 

# 

Partially  relevant 

!  23 

# 

Rel .   or  Part.   rel.  i 

:  28 

# 

Not  relevant  i 

;  39 

# 

Evaluated  ; 

I  67 

# 

Not  evaluated  ; 

0 

# 

of   references  ; 

67 

Overall   precision  ; 

.417 

User  evaluation: 


User's  tiffie 

Dollar  value  assigned 

Worth  assigned 

Problem  resolution 

Satisfaction 


0.17  hrs. 
$10.00 


Searcher  evaluation : 


A: 

#  Relevant 

H: 

Precision 

B: 

#  Partially  relevant 

I: 

Total  #  commands 

C: 

#  Not  relevant 

J: 

Total  #  cycles 

D: 

Total  #  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

L: 

Online  connect  time 

F: 

Total  #  retrieved 

Ms 

Preparation  time 

G: 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

007 

4 

14 

18 

36 

0 

36 

0.643 

0.500 

13 

4 

9 

0.272 

0.200 

0.475 

012 

4 

5 

2 

11 

0 

11 

0.321 

0 . 81 8 

9 

1 

8 

0.150 

0.250 

o.4oe 

023 

d 

1 

2 

3 

0 

3 

0.035 

b  •  333 

8 

1 

8 

b.iba 

0.250 

b.35f 

034 

1 

1 

12 

14 

b 

14 

b .  071 

b.l43 

31 

7 

8 

0.212 

0.083 

0.29E 

036 

b 

b 

0 

b 

b 

0 

b .  bbb 

b .  bbb 

11 

2 

11 

b.265 

0.167 

0.435 

119 

4 

14 

4 

22 

b 

22 

b.643 

0.818 

14 

3 

13 

b.l39 

b .  1 67 

219 

4 

6 

1 

11 

0 

11 

b.357 

0 . 909 

8 

i 

8 

0.076 

0.033 

b.ib'j 

319 

4 

6 

2 

•12 

b 

12 

b.357 

b.833 

7 

2 

8 

0.055 

0.050 

O.IQE 

419 

4 

6 

7 

17 

b 

17 

0.357 

0.588 

11 

2 

18 

0.102 

0.083 

0.16E 

ERIC 
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Question  No.  029 
Database  Noi  068 


******************************* 
«  Sammary  of  Search  Results  * 
****************************** 


# 

Relevant  abstracts  ; 

i  36 

# 

Partially  relevant  i 

I  34 

total 

# 

Reli  or  Parti   rel .  : 

!  70 

# 

Not  relevant  : 

i  80 

# 

Evalaated  i 

!  150 

# 

Not  evaluated  i 

!  77 

Total 

# 

of  references  '. 

:  227 

Overall   precision  ; 

:  .666 

User  evaluation: 


User's  time  s  0.75  hrs. 

Dollar  value  assigned  :  *200.00 

Worth  assigned  :  5 

Problem  resolution  :  4 

Satisfaction  s  ^ 


Searcher  evaluation : 


A: 

#  Relevant 

H: 

Precision 

B: 

#  Partially  relevant 

1: 

Total  #  cdmmshds 

e: 

#  Not  re 1 e vant 

J: 

Total  #  cycles 

E>: 

total  #  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

L: 

Online  connect  time 

F: 

total  #  retrieved 

M: 

Preparation  time 

€: 

Recall 

N: 

Total  time 

SEAR 

A 

B 

e 

D 

E 

F 

G 

Oil 

8 

8 

12 

28 

22 

50 

0 . 229 

027 

15 

13 

50 

78 

23 

101 

0.400 

038 

3 

1 

i 

5 

3 

6 

0.057 

039 

5 

3 

2 

10 

5 

15 

0.114 

040 

3 

0 

0 

3 

0 

3 

0.042 

120 

28 

Id 

13 

51 

24 

75 

0.543 

220 

7 

4 

i 

12 

5 

17 

0.157 

320 

19 

9 

15 

43 

33 

76 

0.400 

42(3 

6 

5 

2 

13 

0 

13 

0.157 

o 

ERIC 


H 

I 

J 

K 

L 

M 

N 

0.571 

32 

2 

24 

0.076 

0.083 

0.15«5 

0.359 

19 

7 

15 

0.161 

0.250 

0.411 

0.800 

17 

6 

9 

0.326 

0.167 

0.492 

0.800 

25 

6 

9 

0.268 

0.250 

0.518 

1 .000 

30 

9 

22 

0.423 

0.333 

0.756 

0.745 

9 

4 

5 

0.198 

0.250 

0.448 

0.917 

7 

2 

13 

0.156 

0.250 

0.406 

0.651 

10 

1 

17 

0.258 

0.283 

0.541 

0.846 

11 

2 

17 

0.188 

0.417 

0.605 

Question  No.  030 
Database  No.  071 

*  Summary  of  Search  Results  * 
********************* 


# 

Relevant  abstracts 

57 

# 

Partial ly  relevant 

25 

Total 

# 

Rel -   or  Part,   rel . 

82 

# 

Not  relevant 

13 

Evaluated 

: 

95 

# 

Not  eva 1 uated 

: 

0 

Total 

# 

of   ref ererices 

: 

95 

Overal 1  precision 

■ 

.865 

User  evaluation: 


User's  time  :   2.00  hrs, 

Dollar  value  assigned  :  none 

Worth  assigned  :  5 

Problem  resolution  :  5 

Satisfaction  :  5 


Searcher  evaluation: 


A: 

#  Relevant. 

H: 

Precision 

B: 

#  Partially  relevant 

I: 

Total  #  commands 

C: 

#  Not' relevant 

J: 

Total  #  cycles 

D: 

Total  #  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

L: 

Online  connect  time> 

F: 

Total  #  retrieved 

M: 

PrejDaration  time 

G: 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

C 

D 

E 

F' 

G 

H 

I 

J 

k 

L 

M 

N 

QD7 

7 

1 

2 

10 

0 

10 

0.097 

0.800 

14 

4 

10 

0.283 

0.066 

0.34 

012 

29 

14 

5 

48 

g 

48 

0.524 

0.896 

12 

1 

10 

0.183 

^0.250 

0.43 

023 

37 

14 

5 

56 

0 

56 

0 . 622 

0.911 

11 

3 

6 

0.183 

0.333 

0.51 

025 

2 

0 

0 

2 

0 

2 

0 . 024 

1  .000 

9 

1 

5 

0.218 

0.250 

0.46 

036 

1 

0 

0 

1 

0 

1 

0.012 

1  . 000 

11 

3 

15 

0.257 

0.483 

£1.74 

120 

32 

16 

5 

53 

0 

53 

0.585 

0.906 

7 

2 

4 

0.056 

0.167 

0.22 

220 

32 

15 

5 

52 

0 

52 

0.573 

0.904 

o 

2 

14 

0.177 

0.183 

0.36 

320 

0 

3 

2 

5 

0 

5 

0.036 

0 . 600 

11 

2 

30 

0.220 

0.250 

0.47 

A20 

A9 

16 

2 

67 

0 

67 

0.793 

0.970 

29 

2 

2 

0.161 

0.066 

0.22 

,3r-5 

o 

ERIC 


Question  No.  031 
Database  No.  061 


*  Summary  of  Search  Results  * 
***************************** 


Total 


Total 


# 

Relevant  abstracts  i 

i  1^ 

# 

Partially  relevant  i 

t  15 

# 

Rel .   or  Part,   rel .  i 

i  29 

# 

Not  relevant  I 

!  85 

# 

Evaluated  i 

!  115 

# 

Not  evaluated  i 

0 

# 

of   references  ; 

!  115 

Overall   precision  ; 

!  .393 

User  evaluation: 


User' s  time 

Dollar  value  assigned 

Worth  assigned 

Prob 1 em  resd 1 ution 

Satisf  action 


0.50  hrs. 
*25.00 
4 


Searcher  evaluation: 


A: 

#  Relevant 

H: 

Precision 

B: 

#  Partially  relevant 

I: 

Total  M  commands 

C: 

#  Not'  relevant 

J: 

Total  #  cycles 

D: 

Total  #  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

L: 

Online  connect  time 

F: 

Total  #  retrieved 

M: 

Preparation  time 

G: 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

C 

D 

E 

F 

G 

H 

i 

J 

K 

L 

H 

007 

1 

0 

19 

20 

0 

20 

0.035 

0.050 

11 

3 

13 

0.558 

0. 

100 

O. 

012 

E 

6 

35 

5i 

0 

56 

0.515 

0 . 261 

lb 

3 

5 

0.151 

0. 

167 

0. 

023 

10 

12 

51 

63 

0 

63 

0.759 

0.359 

32 

6 

28 

1.125 

b. 

750 

1 . 

025 

5 

IS 

27 

0 

27 

0.310 

0 . 333 

7 

2 

3 

0.050 

0. 

250 

0. 

036 

3 

0 

8 

11 

0 

11 

0.103 

0.273 

5 

2 

5 

0.105 

0. 

233 

0. 

120 

7 

5 

23 

35 

0 

35 

0.515 

0.353 

15 

5 

6 

0.199 

b. 

167 

0. 

220 

1 

1 

7 

9 

0 

9 

0 . 069 

0.222 

6 

2 

5 

0.096 

b. 

bS3 

b. 

320 

3 

0 

3 

6 

0 

6 

0.103 

0.500 

9 

1 

17 

0.122 

b. 

b83 

0. 

A20 

A 

2 

3 

9 

d 

9 

0.207 

0.667 

29 

3 

0 

0.307 

0. 

083 

0. 

316 


ERIC 


Question  No.  032 
Database  No.  008 


♦     Summary  of  Search  Results  * 


Relevant  abstracts 

;  113 

# 

Partially  relevant 

:  19 

total 

Rel .   or  Part,    rel . 

;  132 

# 

Not  relevant 

:  IS 

Evaluated 

:  150 

Not  evaluated  i 

!  298 

Total 

# 

of   references  i 

!  448 

Overall  precision 

i  .880 

User  evaluation: 


User's  time  s  2.00  hrs. 

Dollar  value  assigned  :  *2bb.bb 

Worth  assigned  :  5 

Problem  resolution  :  5 

Satisfaction  :  5 


Searcher  evaluations 


A: 

»  Relevant 

H: 

Precision 

B: 

#  Partially  relevant 

I: 

Total  #  commands 

C: 

#  Not  relevant 

J: 

Total  #  cycles 

D: 

Total  ^  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

t.: 

Online  connect  time 

F: 

Total  #  retrieved 

M: 

Preparation  time 

6: 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

C 

D 

E 

F 

G 

OOl 

a 

0 

0 

6 

b 

6 

b.b4B 

008 

«2 

3 

0 

45 

74 

b.34i 

017 

0 

1 

17 

18 

b 

18 

b.bb7 

029 

34 

4 

0 

38 

112 

isb 

b.288 

032 

58 

2 

0 

60 

119 

179 

b.455 

119 

49 

5 

1 

55 

137 

192 

0.409 

219 

50 

5 

i 

56 

142 

198 

b.4i7 

319 

1 

0 

d 

.1 

9 

10 

0.007 

419 

86 

13 

6 

99 

195 

294 

0.750 

H 

i 

J 

K 

b 

N 

1  .bob 

14 

2 

7 

0.274 

0.333 

0.607 

1  iOOO 

31 

12 

31 

0.633 

P.  333 

0.966 

bib55 

9 

3 

9 

0.092 

0.167 

0.25*5 

i  .bbb 

24 

10 

6 

0.744 

0.667 

1  .411 

1  .bob 

7 

3 

6 

0.163 

0.083 

0.246 

b.9S2 

8 

2 

7 

0.151 

0.250 

0.401 

b.9S2 

13 

2 

10 

0.131 

0.250 

0.381 

li  bbb 

10 

2 

17 

0.112 

0.083 

0.195 

1  .obo 

5 

2 

3 

0.085 

0.050 

0.135 

.3  07 

ERIC 


Question  Nb«  033 
Database  Noi  DOS 


*     Summary  of  Search  ResultB  * 


#  Relevant  abstrsicts  :  44 

#  Partially  relevant  :  57 
total  #  Rel-  or  Part-  rel .  :  101 

#  Not  relevant  :  49 
ft  Evaluated  :  150 

#  Not  evaluated  :  123 
Total  #  of  references  :  273 

Overall   precision  :  .673 


User  evaluation: 


User^s  time  :  3.00  hrs. 

Dollar  value  assigned  :  *1000-00 

Worth  assigned  :  5 

Prob 1 em  resbl ution  :  4 

Satisfaction  :  4 


Searcher  evaluation: 


A: 

#  Relevant 

Bi 

Precision 

B: 

#  Partially  relevant 

1: 

Total  #  commands 

e: 

#  Not  relevant 

J: 

Total  #  cycles 

B: 

Total  #  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  eva 1 uated 

L: 

Online  connect  time 

F: 

Total  #  retrieved 

M: 

Preparation  time 

G: 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

e 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

0 

0 

0 

0 

G 

D 

D.DDO 

D.DDO 

22 

5 

10 

6.675 

C.833 

1.5C 

ooa 

Q 

0 

3 

3 

2 

5 

Q.QOO 

0.000 

16 

4 

39 

0.450 

0.?:50 

0.7C 

Q17 

3 

G 

0 

3 

4 

7 

0 . 029 

1  .  ODD 

17 

4 

6 

0.324 

0 . 667 

0.9*; 

Q29 

1 

Q 

0 

1 

1 

2 

0.009 

1  .000 

8 

2 

3 

0.192 

0.417 

0.6C 

032 

15 

26 

6 

47 

38 

as 

0.406 

0 . 872 

13 

4 

6 

0.217 

0.083 

0.30 

119 

20 

12 

15 

47 

24 

71 

0.317 

0 . 681 

20 

6 

9 

0.270 

0 . 333 

0.6C 

219 

7 

17 

12 

36 

41 

77 

0 . 238 

0.667 

15 

5 

11 

0.300 

0.167 

b.4£ 

319 

4 

A 

14 

14 

28 

0.099 

0.714 

10 

3 

10 

0.143 

0.250 

0.39 

419 

6 

6 

11 

23 

15 

38 

0.119 

0.522 

11 

2 

7 

0.196 

0.167 

0.36 

erJc 


Qaes^ion  Noi  034 
ba-babasi  No,  013 


*     Summary  of  Search  Resaits  * 


#  Relevant  abstracts  : 

!  10 

#  Partially  relevant  : 

!  39 

total 

#  Rsl.  or  Part,   rel . 

!  49 

#  Not  relevant 

!  ISO 

#  Evaluated  ; 

!  149 

#  Not  evaluated  : 

!  207 

Total 

#  of   references  ; 

!  356 

Overall  precision  : 

!  i328 

User  evaluations 


User's  time    :  3.00  hrs. 

Dollar  value  assigned  :  *150.OO 

Worth  assigned  :  4 

Problem  resolution  :  3 

Satisfaction  :  4 


Searcher  evaluation : 


A: 

#  Relevant 

H: 

Precision 

B: 

#  Partially  relevant 

I: 

Total  #  commands 

e: 

#  Not  relevant 

J: 

Total  #  cycles 

b: 

Total  S  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

t: 

Online  connect  time 

F: 

Total  #  retrieved 

M: 

Preparation  time 

G: 

Recal 1 

N: 

Total  time 

e         E»         E         F  G 

H             I         J  K 

M 


N 


2 
12 
4 
13 
1 
7 
1 
5 
0 


o 

70 
5 
7 
Q 

14 
3 

10 
4 


<3 

86 

9 
21 

1 
24 

4 
17 

4 


0 

186 

i 

O 

0 
10 
7 
8 
9 


3  0.061 
>72  0.327 

10  0.081 
21  0.286 

1  0.020 
34  0.204 

11  0.020 
25  0.143 
13  0.000 


1  .ODD 
0.186 
0.444 
0.667 
1.000 
0.417 
0.250 
0.412 
0.000 


12 
15 
15 

5 
12 

5 
13 
13 

9 


2 
6 
5 
2 
5 
1 
2 
2 


7  0.278 
e  0.426 

6  0.333 
0.069 
0.211 
0.167 
0.293 

10  0.203 

7  0.288 


5 
6 
3 
7 


0.583 
0 . 1 33 
0  ■<  333 
0.417 
0.083 
0.250 
0.083 
0 . 066 
0.083 


0.86 
0.55< 
0.66< 
0.48< 
0.29' 
0.411 
b.37< 
d .  26* 
0.371 
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Question  No.  035 
Database  Nd«  154 


fr****************************** 
*     Summary  of  Search  Results  * 


#  Relevant  abstracts 
«  Partially  relevant 

Total  S  Rel-   or  Part-   rel - 

#  Not  re 1 e vant 

#  Evaluated 

#  Not  evaluated 
Total  #  of  references 

Overal  1  jsrecision 


31 
20 
51 
14 
65 
0 
65 
-784 


User  evaluation: 


User^'s  time  :   1-50  hrs. 

Dollar  value  assigned  :  $25-00 

Worth  assigned  :  5 

Prob 1 em  resb 1 ut ion  :  4 

Satisi action  :  4 


Searcher  ©valuations 


A: 

»  Relevant 

H: 

Precision 

B: 

»  Partially  relevant 

I: 

Total  #  commands 

C: 

»  Not  relevant 

J: 

Total  ttvCycles 

D: 

Total  #  evaluated 

K: 

Total  #  search  terms 

E: 

»  Not  evaluated 

L: 

Online  connect  time 

F: 

Total  #  retrieved 

M: 

Preparation  time 

G: 

Recal 1 

N: 

Total  time 

SEAP 

A 

B 

C 

D 

E 

F 

G 

H 

i 

j 

k 

t 

M 

N 

001 

1 

0 

6 

1 

d 

1 

0 . 01 9 

1 .  odd 

•16 

5 

4 

0i445 

0 . 333 

0.778 

008 

6 

5 

5 

16 

0 

16 

0 . 21 6 

0.688 

i2 

6 

20 

0  i  361 

0.200 

0.561 

017 

1 

0 

0 

1 

d 

1 

0.019 

1  .ddd 

6 

1 

3 

diOSO 

0 . 083 

0.163 

029 

12 

4 

6 

22 

0 

22 

0.314 

d .  727 

14 

3 

Id 

0.219 

0.250 

0.469 

032 

9 

10 

3 

22 

d 

22 

0.373 

D.864 

15 

5 

6 

d.l84 

0.083 

0.267 

120 

1 

0 

5 

b 

3 

d .  098 

i  .Odd 

7 

3 

5 

0.132 

0.133 

0.265 

220 

1 

0 

5 

d 

5 

d .  098 

i .  ddd 

7 

2 

4 

0.127 

0.167 

0.294 

320 

1-* 

6 

0 

20 

0 

20 

0.392 

i  .ddo 

19 

6 

22 

0.556 

0.117 

0.673 

420 

4 

2 

1 

7 

d 

7 

0.118 

0.857 

7 

1 

4 

0.095 

0.167 

0.262 

erJc 
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Quesibibh  No.  036 
Database  No.  09D 

♦Summary  of  Search  Results  * 


#  Relevant  abstracts  s  62 

#  Partially  relevant  :  49 
Total   #  Rel .   or  Part,   rel .  :  111 

#  Not  relevant  :  39 

#  Evaluated  :  150 

#  Not  evaluated  :  57 
Total  #  of   references  :  207 

Overall   precision  :  .740 


User  evaluation: 


User's  time  :  2.00  Mrs. 

Ddl  lar  Value  assigned  :  $50 . 0C3 

(^drth  assigned  :  4 

Problem  resolution  :  3 

Satisfaction  :  4 


Searcher  eva 1 uation s 


A: 

#  Relevant 

H: 

Precision 

B: 

#  Partially  relevant 

I : 

Total  #  commands 

Cs 

#  Not  relevant 

Js 

Total  #  cycles 

D: 

Total  #  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

Ls 

Online  connect  time 

F: 

Total  #  retrieved 

Ms 

Preparation  time 

G: 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

e 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

BC37 

13 

13 

11 

37 

36 

73 

0 . 234 

0.703 

25 

7 

13 

0.341 

0.167 

0.50S 

di2 

5 

2 

5 

12 

d 

12 

0.063 

Q.BS3 

13 

2 

13 

0.213 

0.417 

0.63Q 

023 

20 

17 

13 

50 

12 

62 

0 . 333 

0.740 

23 

5 

12 

0.597 

0.583 

1  .180 

Q2S 

31 

13 

12 

56 

b 

56 

0 . 396 

b.786 

9 

i 

8 

0.111 

0i250 

0.361 

□36 

28 

9 

10 

47 

b 

47 

0 . 333 

0.787 

7 

2 

4 

0.157 

0.083 

0.240 

120 

6 

9 

i 

16 

9 

25 

0.135 

0 . 938 

10 

3 

15 

0.153 

0ill7 

0.270 

220 

4 

2 

0 

6 

0 

6 

0.054 

i  .000 

6 

i 

14 

0.126 

0ill7 

0.243 

320 

7 

3 

i 

ii 

2 

13 

0.090 

0.909 

7 

i 

9 

b.bsi 

0.083 

b.l64 

420 

4 

i 

0 

5 

b 

5 

0.045 

1  .000 

28 

2 

1 

0.200 

0.083 

0.283 
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Question  No.  037 
Database  Nb«  016 

*     Sammary  of  Search  Results  * 


Relevant,  abstracts 

7S 

Partially  relevant 

!  -IS 

Reli   or  Part,   rel . 

:  96 

4* 

Not  relevant 

:  54 

Evaluated 

8  15P 

Not  evaiaated 

J  69 

of  references 

:  219 

Gverall  precision 

:  .640 

User  evaf uation : 


User's  timb  :  3.DD  hrs. 

Dollar  value  assigned  :  *300.00 

Worth  a^jsigried  :  5 

Problem  resolution  :  4 

Satisfaction  s  5 


Searcher  evaluations 


A: 

#  Relevant 

H: 

Precision 

B: 

#  Partially  relevant 

I : 

Total  #  commands 

C: 

#  Not'  relevant 

J: 

Total  »  cycles 

D: 

Total  #  evaluated 

K: 

Total  #  search  terms 

E: 

#  Not  evaluated 

Ls 

Online  connect  time 

F: 

Total  #  retrieved 

M: 

Preparation  time 

Recal 1 

N: 

Total  time 

SEAR 

A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

015 

11 

3 

33 

47 

25 

72 

0.146 

0 . 298 

26 

5 

7 

6.733 

0.500 

i  .22 

030 

3 

0 

2 

5 

0 

5 

0.031 

0.600 

39 

9 

17 

0.869 

0.333 

1 .2e 

034 

16 

1 

6 

23 

2 

25 

0 . 177 

0.739 

25 

4 

7 

0 . 220 

0.167 

b.3£ 

035 

29 

1 

6 

36 

19 

55 

0  .313 

0.833 

17 

5 

9 

0.252 

0.0S3 

0.32 

041 

31 

6 

ID 

47 

26 

73 

0.385 

0.787 

13 

4 

7 

0.198 

0.167 

0.3£ 

119 

20 

2 

0 

22 

0 

22 

0.229 

1  .000 

16 

3 

9 

0.1S8 

0.133 

0.29 

219 

19 

1 

1 

21 

0 

21 

0.208 

0.952 

11 

3 

9 

0.148 

0.033 

o.ia 

319 

18 

1 

1 

20 

0 

20 

0.198 

0.950 

13 

4 

8 

0.133 

0.016 

bii4 

419 

34 

10 

5 

49 

0 

49 

0.458 

0.898 

29 

5 

11 

0.490 

0.250 

0.74 

ERIC 


Qaestlon  No^  038 
Database  No^  061 


*     Summary  of  Search  Resu Its  * 


Total 


Total 


# 

Relevant  abstracts  : 

!  79 

# 

Partial iy  relevant  i 

!  29 

# 

Rei.  or  Part,   rel .  i 

!  iOS 

# 

Not  relevant  i 

!  42 

Evaiaated  i 

I  i^d 

# 

Not  evaiaated  i 

I  595 

# 

of  references  ! 

!  745 

Overall   precision  : 

.720 

User  eva  1  aat ion 


User's  time             _  __   ■  *^lOD  hrs. 

Worth^ssigned  ■  ^ 

Problem  resolution  :  5 

Satisfaction  :  5 


Searcher  evaluation: 


A: 

#  Relevant 

H: 

Precision 

B: 

#  Partially  relevant 

I: 

Total  #  commands 

e; 

#  Not  relevant 

J: 

Total  #  cycles 

£»: 

Total  #  evaiaated 

K: 

Total  #  search  terms 

E: 

#  Not  evaiaated 

b: 

Online  connect  time 

F: 

Total  #  retrieved 

M: 

Preparation  time 

6: 

Recall 

N: 

Total  time 

e         D         E         F  G 

H             1         J  K 

B 

5  3  16  53  69  0.120  0.813  24  3 

9  19  34  24d  274  0.139  D.441  20  5 

1  4  13  32  45  0.083  D.692  32  6 

2  2  10  15  25  0.074  D.SOD  19  4 
8  3  40  64  104  0.343  0.925  45  14 

1  1  7  30  37  0.055  0.S57  24  7 

2  2  12  71  83  0.092  0.833  30  7 
2  4  9  37  46  0.046  0.556  23  5 
G  -4  15  79  94  0.102  0.733  50  7 


16  0.361 
10  0.312 
4  0.3Q3 
0.278 
0.857 
0.469 
31  0.352 
49  0.455 
4  0.811 


13 
17 

14 


M 

0.500 
0.417 
0.083 
0.083 
0.417 

0.267 
0.250 
0.167 
0.083 


N 

0.861 
0.72S 
0.38<f 
0.361 
1.27^ 
0.73i 
0.602 
0.622 
0.89^ 
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Question  No.  03? 
Database  No.  ni5 

*     Sufnmary  of  Search  Resu  1  ts  * 


Relevant  abstracts 

!  26 

Partially  relevant 

:  3S 

total 

m 

Rel .  or  Part,   rel .  i 

!  hit 

i^Qt  relevant  i 

!  102 

Evaluated  ; 

!  16^ 

Not  evaluated  t 

!  a 

Total 

of   references  ! 

I  166 

Overall   precision  : 

'  .386 

User  evalLtation; 


User's  time  :  2-50  hrs. 

Dollar  value  assigned  :  $100.00 

Worth  assigned  :  ^ 

Problem  resolution  :  ^ 

Satisfaction  :  4 


Searcher  evaluation: 


A: 

#  Relevant 

H: 

Precision 

B: 

#  Partially  relevant 

I: 

Total  #  commands 

C: 

8  Not  relevant 

J: 

Total  #  cycles 

D: 

Total  #  evaluated 

K: 

Total  #  search  terms 

E: 

g  Not  evaluated 

L: 

Online  coririect  time 

F: 

Total  #  retrieved 

M: 

Preparation  time 

G: 

Reca 1 1 

N: 

Total  time 

SE^R 

A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

015 

0 

0 

0 

0 

0 

D 

0.000 

0.000 

10 

3 

10 

0.166 

0.500 

0 .  h&i 

030 

4 

3 

7 

14 

0 

14 

0.109 

0.500 

13 

3 

12 

0.291 

0.583 

O.S7i 

034 

2 

5 

5 

12 

0 

12 

0.109 

0.583 

17 

4 

7 

0.183 

0.083 

0.26i 

035 

0 

0 

6 

6 

0 

6 

O.OCO 

0.000 

16 

5 

12 

0 . 31 7 

0 . 333 

0.65C 

041 

1 

0 

1 

2 

0 

2 

0.015 

0.500 

19 

4 

7 

0.285 

0.417 

0.702 

119 

0 

0 

4 

4 

0 

4 

0.000 

0.000 

18 

4 

9 

0.173 

0 . 1 67 

0.34C 

219 

15 

6 

6 

27 

0 

27 

0.328 

0.778 

13 

4 

7 

0.096 

0.083 

D.17«3 

319 

5 

22 

56 

83 

0 

83 

0.422 

0.325 

10 

1 

19 

0.192 

0.117 

O.30«5 

419 

6 

6 

26 

38 

0 

38 

0.188 

0.316 

6 

1 

5 

0.082 

0.333 

0.41E 

ERIC 
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# 

Relevant  abstracts  s 

i  77 

Partially  relevant  ! 

i 

Total 

# 

Rel.  or  Part,   rel .  ! 

:  117 

« 

Not  relevant  ; 

!  32 

« 

Evaluated  ; 

:  15? 

# 

Not  evaluated  ! 

!  331 

Total 

of  references  : 

560 

Overall  precision  ; 

•  .785 

User  evaluation: 


User's  time  :  4.00  hrs. 

Dollar  value  assigned  :  *400.00 

Worth  assigned  :  4 

Problem  resolution  :  4 

Satisfaction  :  4 


Searcher  evaluation : 


At  #  Relevant  H: 

B:  #  Partial ly  relevant  Is 

Cs  #  Not  relevant   J: 

bs  Total  #  evaluated  K: 

E:  #  Not  evaluated  L.: 

F:  Total  #  retrieved  M: 

6:  Recall  N: 


Precision 

Tota 1  #  commands 

Total  #  cycles 

Tota 1  *  search  terms 

Online  connect  time 

Preparation  time 

Total  time 


B 

e 

D 

E 

F 

G 

H 

I 

J 

K 

L 

M 

N 

6 

D 

16 

61 

77 

0.137 

1 

.666 

32 

9 

8 

6.AA3 

b;25Q 

6.693 

A 

32 

38 

A 

A2 

D.P51 

0 

.158 

13 

3 

15 

0.293 

0;500 

0.793 

2 

0 

8 

7 

15 

0.068 

1 

.000 

32 

8 

10 

0.367 

0;25Q 

0.617 

A 

0 

15 

A7 

62 

0.128 

1 

.000 

lA 

6 

6 

0.257 

0;083 

0.3A0 

9 

0 

25 

18 

A3 

Q.21A 

1 

.000 

19 

7 

3 

0 . 321 

0. 555 

0.65A 

2 

0 

10 

20 

30 

0.085 

1 

.000 

17 

5 

5 

0.303 

0.500 

0.805 

9 

0 

21 

52 

73 

0il79 

1 

.000 

20 

5 

20 

0.38A 

0.167 

0.551 

12 

0 

50 

189 

239 

0.A27 

1 

.000 

29 

7 

31 

0.398 

0 . 250 

0.6A8 

Q 

0 

3 

A 

7 

0.025 

1 

iOOO 

9 

2 

3 

0.1A3 

0.333 

0.A76 
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APPENDIX       StikMffit  OF  SEARCH  KBISULTS  OBTAiNED  BY  OUTSIDE  SEARCEBSS 


Appendlx^C  contains  identical  information  as  part  of  the  

information  presented  in  Appendix-R  (combined  search  results  for  the 
questions  and  search  results  itemized  by  the  five  outside  searches) 
except  that  this  time  the  data  are  compiled  by  searcher  number 
Instead  of  by  question  number*    Each  consecutive  page  lists  in 
searcher  number  order  results  for  that  individua^searcher.  Within 
the  compilation  for  each  searcher  are  iited  in  number  order  the 
questions  seached  by  that  searcher.    Appendix  6  includes  the 
following: 

i*  Searcher  number  of  the  36  outside  searchers  coded  as  60i  - 
017,  021,  and  023  -  041. 

2.  Question  number  and  database  number  of  the  questions  assigned 
to  that  individual  searcher*    The  first  three  digits  are  the 
question  number  and  the  second  three  digits  are  the  database 
number • 

3*  The  combined  results  of  all  nine  searches  for  each  question 
done  by  the  searcher  including: 

Ai  Total  number  of  relevant  items    

B*  Total  number  of  partially  relevant  items 

Cm  total  number  of  not  relevant  items 

D*  total  number  of  evaluated  items  

E*  total  number  of  not  evaiuate^items,  and 

F*  total  number  of  retrieved  items 

4*  Individual  search  results  for  that  question  including: 

A*  Nmber  of  relevant^ items  obtained  by  searcher  numbcsr  

B*  Number  of  partially  relevant  items  obtained  by  searcher 

nimber   

C*  Number  of  not  relevant  items  obtained  by  searcher  number 


bi  Number  of  items  retrieved  by  searcher  number  which 

were  evaluated 

E*  Number  of  items  retrieved  by  searcher  number    which 

were  no t_ evaluated  

F.  total  number  of  items  retrieved  by  searcher  number   

5*  The  effectiveness  measures  for  that  question  including: 

Am  Search  recall  

B.  Search  precision  ^ 

Cm  Number  of  search  terms  used  by  searcher  number   

b*  Number  of  cotmands  used  by  searcher  number   

E*  Number  of  cycles  used  by  searcher  nt^ber  

fm  Online  connect  time  used  by  searcher  number   

6i  Offline  preparation  time  used  by  searcher  number   

H.  total  search  time  used  by  searher  number   
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Searcher  No.  001 


QUEST- DBASE 
NUMBER 

G32D0S 


TOTAt  # 
RELV 

•113 


TOTAt  # 
PREt 

19 


TOTAt  # 
NREt 

IS 


TOTAt  # 
EVAt. 

150 


TOTAt  # 
NEVt 

298 


TOTAL  # 
REtRlEVEt 


#  of   relevant  items 

#  of  partially  relv 

#  of     not  relevant 


total  #  evaluated 
#     not  evaluated 


6 
0 
0 


O 


total  #  retrieved  :  6 

recall  :  0.D45 

precision  :  liSOO 

total  #  of  search  terms:  7 
total  #  of  commands  :  14 
total  #  of  cycles  :  2 


on-line  connect  time 
off  preparation  time 


s  0.274 
:   0  ^  333 


total   searching  time 


0.607 


033008 


44 


57 


49 


150 


123 


273 


#  of   relevant  items  s  0 

#  of   partially  relv  :  0 

#  of     not  relevant  :  0 

total  #  evaluated  :  0 

#  not  evaluated  :  0 

total  #  retrieved  :  0 

recall  :  D.OOD 

precision  :  0.000 

total  #  of  search  terms:  10 

total  #  of  commands  :  22 

total  #  of  cycles  :  5 

bh— line  cbhhect  time  :  0.675 

off  preparation  time  :  D.S33 

total  searching  time  :  1.508 
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Searcher  No.  001 


QUEST-DBASE       TOTAL  «       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVEC 

019075  27  49  74  150  52  202 


#  of   relevant  items 

■ 

□ 

#  of   partially  reiv 

■ 

□ 

#     of     not  relevant 

s 

1 

tota 1  #  eva 1 uated 

■ 

1 

#     not  evaluated 

■ 

i 

tota 1  #  retrieved 

■ 

2 

recal 1 

■ 

D.bod 

precision 

■ 

□  .□do 

total   #  of   search  terms 

■ 

a 

total  #  of  commands 

■ 

16 

total  #  of  cycles 

■ 

2 

on— 1 ine  connect  time 

s 

0.305 

of  f   preparation  time 

■ 

□  .417 

tota 1   search ing  time 

■ 

0.722 

A3  81 

150 

#  of 

rel evant  items 

■ 

0 

#  of 

pairtiai  ly  reiv 

■ 

i 

#  of 

not  relevant 

s 

i 

total 

#  evaluated 

■ 

2 

#     not  evaluated 

■ 

2 

total 

#  retrieved 

s 

4 

recal 1 

m 
m 

0 

.014 

precision 

m 
m 

0 

.500 

total 

#  of   search  terms 

n 
m 

4 

total 

#  of  commands 

m 

13 

total 

#  of  cycles 

S 

2 

on-i ine  connect  time 

0 

.239 

off   preparation  time 

■ 

0 

.  333 

total 

searching  time 

■ 

d 

.572 
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Searcher  No.  001 


QUESt-bBASE       TdtAt  #       TOTAt  #       tOTAL  #       TOTAL  #       tOTAL  #       TOTAL  « 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVED 

034013  Id  39  100  149  207  356 


#  of 

relevant  items 

i 

#  of 

partial ly  relv 

2 

#  of 

not  relevant 

0 

total  #  evaluated 

s 

3 

#     not  evaluated 

■ 

H 

0 

total 

#  retrieved 

s 

3 

recai 1 

:  0.061 

precision 

:   1 . 000 

total 

#  of  search  terms 

s 

7 

total 

#  of  commands 

m 

12 

total 

#  of  cycles 

S 

2 

on- 1 ine  connect  time 

:  0.278 

of f   preparation  time 

:  0.583 

total 

searching  time 

:  0.861 

Q3515A  31  2G  .  65  0  65 


#  of   relevant  items         :  1 

#  of   partially  relv         :  □ 

#  of     not  relevant         :  □ 


total   **^yaiuated  :  i 

#     not  evaluated  :  □ 


total  #  retrieved  :  1 

recall  :  QiQi9 

precision  :  i.QGO 

total  #  of  search  terms:  4 

total   #  of  commands  :  16 

total  #  of  cycles  :  5 

on-1 ine  connect  time  :  0-445 

off   preparation  time  :  Gi333 


total   searching  time       &  S i77S 
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Searcher  No.  002 


QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVED 

bdibli  27  46  75  148  626  774 


#  of 

relevant  items  : 

3 

#  of 

partial ly  relv  : 

2 

#  of 

not  relevant  : 

i 

total 

#  evaluated  : 

6 

#     not  evaluated  : 

9 

total 

#  retrieved  : 

15 

recall  : 

0.068 

precision  : 

0.833 

total 

#  of  search  terms: 

13 

total 

#  of  commands  : 

21 

total 

#  of  cycles  : 

5 

on- 1 i  ne  connect  t i  me  s 

0.167 

off   prepar  stion  time  : 

0.250 

total  searching  time       :  Q.417 


003064  36  47  68  151  272  423 


#  of 

re 1 evant  i terns 

9 

#  of 

partial ly  relv 

18 

#  of 

not  relevant 

■ 

26 

total 

#  evaluated 

53 

#     not  evaluated 

10 

total 

#  retrieved 

■ 

63 

recal 1 

0.325 

precision 

0.509 

total 

#  of  search  terms 

13 

total 

#  of  commands 

14 

total 

#  of  cycles 

5 

bn-1 irie  connect  time 

s 

0.167 

off   preparation  time 

0.250 

total   searching  time       :  0.417 
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Searcher  No.  0O2 


QUEST- DBASE 
NUMBER 

£309037 


tdtAt  # 
RELV 

•IS 


TOTAt  # 


TOTAL  # 


TOTAL  # 


PREL  NREL 

EVAL 

• 

4S  84 

150 

#  of  relevant  items 

s 

i 

#  of  partial  iy  reiv 

s 

6 

#     of     not  relevant 

s 

2 

tota 1  #  eva 1 aated 

■ 

9 

#     not  evaluated 

s 

45 

total  #  retrieved 

. 

54 

recai 1 

■ 
■ 

0.106 

precision 

m 
m 

0.778 

total  #  of  search  terms 

m 
m 

IS 

total  #  of  commands 

m 
m 

24 

total  #  of  cycles 

m 
m 

6 

on-i ine  connect  time 

S 

0 . 273 

off  preparation  time 

S 

0.250 

total  searching  time 

■ 
m 

0.523 

TOTAL  # 
NEVL 

495 


TOTAL  S 
RETRIEVED 

645 


016011 


25 


37 


108 


170 


170 


#  of  relevant  items 

#  of  partially  reiv 

#  of     not  relevant 


20 
21 
3b 


total  #  evaluated 
#     not  evaluated 


71 

b 


total 

#  retrieved 

m 
m 

71 

recai  i 

m 
m 

0.661 

precision 

m 
m 

b.577 

total 

#  of   search  terms 

m 

3 

total 

#  of  commands 

m 

6 

total 

#  of  cycles 

m 
m 

2 

on-line  connect  time 

m 

O.bbj 

of  f   preparation  time 

m 

0.167 

total  searching  time 


0.225 


Searcher  No;  002 


QUEST- DBASE       TOtAL  #       TOTAl.  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  REtV  PREL  NREL  EVAL  NEVL  RETRIEVED 

021037  19  6  77  102  0  102 


ERIC 


PREt  NREL 

EVAL 

6  77 

102 

#  of   relevant  items 

s 

#  of  partial ly  relv 

• 

0 

#     of     not  relevant 

s 

9 

tota 1  #  eva 1 uated 

s 

15 

#     not  evaluated 

■ 

m 

d 

total  #  retrieved 

m 

15 

recal 1 

m 

D .  240 

preciBion 

■ 
■ 

d .  400 

total  #  of  search  terms 

m 

3d 

total  #  of  commands 

m 

30 

total  #  of  cycles 

m 

m 

7 

on-i  ine  connect  time 

S 

d .  387 

of  f  preparation  time 

S 

d .  250 

tota 1  search ing  time 

S 

d .  637 

Q250Q1  30  26  94  150  430  580 


#  of 

relevant  items 

s 

16 

#  of 

partial ly  relv 

s 

15 

#  of 

not  relevant 

s 

19 

total 

#  evaluated 

■ 
• 

50 

not  evaluated 

■ 

199 

total 

#  retrieved 

m 
m 

249 

recal 1 

S 

0.554 

precision 

m 
m 

0.620 

total 

#  of   search  terms 

m 

19 

total 

#  of  commands 

m 

22 

total 

#  of  cycles 

m 

5 

on- 1 ine  connect  time 

m 

0.359 

of  f  preparation  time 

m 

0 . 333 

total 

se a rch ing  t i me 

m 

0.692 
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Searcher  No.  003 


QUEST-DBASE 
NUMBER 

002218 


005148 


RELV 

37 


16 


#       TOTAL  #       TOTSL  # 

TOTAL 

PDpl  tMPPI 

C  V  rlL. 

36  1S6 

229 

#  of   relevant  xtems 

■ 

5 

#  of   partially  relv 

• 

5 

#     of     not  relevant 

S 

12 

total  #  evaluated 

• 

22 

#     not  evaluated 

• 

• 

0 

total  #  retrieved 

• 
• 

recal 1 

• 

0.137 

precision 

S 

0.455 

total  #  of  search  terms: 

S 

total  #  of  commands 

• 
• 

13 

total  #  of  cycles 

■ 

4 

on— line  connect  time 

S 

0.173 

off   preparation  time 

u 

0  .333 

total   searching  time 

S 

0.506 

23  48 

87 

#  of   relevant  items 

u 
m 

7 

#  of  partially  relv 

m 

15 

#     of     not  re 1 evant 

m 

11 

tota 1  #  eva 1 uated 

m 

33 

#     not  evaluated 

m 

m 

0 

total  #  retrieved 

m 
m 

33 

recal 1 

m 
m 

0.564 

precision 

m 

n  -  AA7 

w  .  WW  f 

total  #  of  search  terms 

m 

4 

total  #  of  commands 

m 

10 

tota 1  #  of  eye 1 es 

m 

4 

bn-1 i^e  connect  time 

m 

0.123 

of  f   preparation  time 

m 

0.167 

tota 1   sea rch i ng  time 

m 

0.290 

TOTAL  # 
NEVL 

O 


TOTAt  # 
RETRiEVEb 

229 


88 
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Searcher  No.  003 


QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETR±EVE[ 

007075  70  39  40  149  336  485 


«  of 

relevant  items 

s 

1 

#  of 

partial ly  rely 

s 

3 

#  of 

riot  relevant 

— 

1 

total 

#  evaluated 

■ 

5 

#     not  evaluated 

■ 

0 

total 

#  retrieved 

■ 

5 

reouil  1 

s 

0.036 

precision 

■ 

0.800 

total 

#  of  search  terms 

0 

8 

total 

#  of  commands 

m 
m 

8 

total 

#  of  cycles 

m 

2 

on- 1 ine  connect  t i me 

S 

O.IOS 

off   preparation  time 

■ 

0.250 

total 

searching  time 

0.358 

013015  7  36  106  149  14  163 

#  of   rel eyant  items  :  3 

#  of   partially  relv  :  5 

#  of     not  relevant  :  22 

tota 1  #  eya 1 uated  :  30 

#  riot  eya 1 uated  :  3 

total  #  retrieved  :  33 

recall  :  0.186 

precision  :  0.267 

total  #  of   search  terms:  18 

total  #  of  commands  :  29 

total  #  of  cycles  s  6 

on-1 ine  connect  time  :   0 . 404 

off  preparation  time  :  0.417 

total   searching  time  :   0. 821 
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Searcher  No.  DOS 


QiJESt-bBASE 
NUMBER 

014111 


018015 


RELV 
35 


66 


#       TOTAL  #       TOTAL  # 

TOTSL 

PREL  NREL 

EVSL 

71  51 

157 

#  of   re  1  evantr  i temB 

s 

16 

#  of  partially  relv 

26 

#     of     not  re 1 evant 

s 

9 

total  #  evaluated 

■ 



51 

#     not  evaluated 

■ 
■ 

1 

total  #  retrieved 

s 

52 

recal 1 

s 

d .  396 

precision 

s 

0.824 

tota 1  #  of  search  terms : 

11 

total  #  of  commands 

s 

15 

tota 1  #  o f  eye 1 es 

■ 

N 

2 

on— 1 ine  connect  time 

m 
m 

0.314 

off  preparation  time 

m 

0.583 

tota 1   search i ng  t i me 

m 
m 

0.897 

38  46 

150 

#  of  relevant  items 

S 

6 

#  of  partial ly  relv 

m 

d 

#    of     not  relevant 

m 

m 

0 

tota 1  #  eva 1 uated  , 

S 



0 

#     not  evaluated 

a 
■ 

1 

total  #  retrieved 

S 

1 

recall 

0.000 

precision 

0 . 000 

total  #  of  search  terms 

S 

24 

total  #  of  commands 

S 

28 

total  #  of  cycles 

m 

7 

on-1 ine  connect  time 

S 

0.438 

off  preparation  time 

0 . 333 

total  searching  time 

■ 
• 

d .  771 

TOTSL  # 
NEVL 


TOTAb  # 
RETRIEVED 

159 


613 


763 


3^5 


Searcher  No.  004 


QUEST-DBASE 
NUMBER 

002216 


005148 


RELV 
37 


16 


#       TOTAL  #       TOTAb  # 

TOTAb 

pppi  Kippi- 
1  net-  iNncc^ 

CVHC 

36  156 

229 

#  of        levant   i teni^ 

■ 

□ 

#  of  partially  rely 

s 

0 

#     of     not  re 1 evant 

s 

2 

tota 1  #  eva 1 Uated 

s 

2 

#     hot  evaluated 

s 

0 

total  #  retrieved 

s 

2 

reca 1 1 

s 

C3-goo 

j3recision 

s 

n .  nnn 

u  ■  t_ILJ  LJ 

total  #  of  search  terms: 

15 

total  #  of  commands 

s 

IS 

tota  1    #  CDf    cvc  1  es 

3 

bn-1 ine  connect  time 

s 

6.148 

wXl     )JLc7fJinLini«rXwll  Willis 

s 

0.100 

tota 1  search i ng  t i me 

s 

0.248 

23  48 

87 

#  of   relevant  items 

a 

M 

•1 

#  of   partial ly  rel v 

s 

2 

#     of     not  relevant 

■ 
m 

4 

total  #  evaluated 

■ 

7 

#     not  evaluated 

s 

b 

total  #  retrieved 

■ 

7 

recal 1 

m 
m 

0.076 

precision 

B 

■ 

0  .  429 

tota 1  #  o f  sea rch  te  rms 

B 

4 

tota 1  #  of  command s 

m 

11 

total  #  of  cycles 

B 

■ 

3 

on-i ine  connect  time 

S 

0.202 

off  preparation  time 

« 

0 . 083 

tota 1  sea rch  i  ng  t i  me 

it 
• 

0 . 285 

TOTAb  # 
NEVL 


tOtAb  # 
RETRiEVEb 

229 


88 
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Searcher  No.  004 


QUEST-DBASE 
NUMBER 


007075 


TOTAb  # 
RELV 

70 


TOTAt  # 
PREt 

39 


TOTAL  # 
NREL 

40 


TOTAL  # 
EVAL 

149 


TOTAL  # 
NEVL 

336 


_ TOTAL  # 
RETRIEVEi: 

485 


#  of   relevant  items 

#  of   partially  reiv 

#  of     not  relevant 


16 
23 
28 


total  #  evaluated 
#     not  evaluated 


67 
278 


total  #  retrieved 


s  34S 


reca 1 1 
precision 


0.358 
0.582 


total  #  of  search  terms:  14 
total  #  of  commands  :  24 
total  #  of  cycles  s  6 

on- line  connect  time  :  Qi44l 
off  preparation  time       :  0.167 


total  searching  time 


0.608 


013015 


36  1Q6 

149 

#  of 

relevant  items 

s 

1 

#  of 

partial ly  rel v 

s 

1 

#  of 

not  relevant 

8 

0 

total 

#  evaluated 

S 

2 

#     not  evaluated 

S 

0 

total 

#  retrieved 

■ 
■ 

2 

recal 1 

• 
■ 

0.046 

precision 

■ 
■ 

1  .000 

total 

#  of  search  terms 

S 

11 

total 

#  of  commands 

17 

total 

#  of  cycles 

S 

3 

bn-1 ihe  connect  time 

5 

D.ISS 

of  f  preparation  time 

S 

OilSO 

total 

sea  rch  i  ng  t i  me 

S 

0.308 

14 


163 
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Searcher  No.  004 


QUEST-DBASE       TOTAL.  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVEC 

□14151  35  71  51  157  2  l59 


#  of   relevant  x terns 

s 

IQ 

#  of   partial iy  reiv 

s 

15 

s 

m 

tota 1  #  eva 1 oated 

■ 

30 

Tnf"!'^     o  w  a  ^  r  1 A 
\         liOX/    C7  V  a  X  uta  OcrU 

s 

JL 

tctal  #  retrieved 

■ 

31 

recal 1 

s 

0.236 

precision 

s 

0.833 

total  #  of  search  terms 

« 
■ 

11 

total  •»?  of  commands 

■ 
m 

18 

total  #  of  cycles 

■ 

2 

on-1 ine  connect  time 

s 

0.248 

of  f  preparation  time 

■ 

0.233 

total   searching  time 

• 

0.481 

38  A6 

150 

#  of 

relevant  items 

s 

55 

#  of 

partial ly  reiv 

s 

29 

#  of 

not  relevant 

s 

31 

total 

#  evaluated 

s 

115 

#     not  evaiciated 

s 

541 

total 

#  retrieved 

s 

656 

reca 1 1 

. 

0.808 

precision 

. 

0.730 

total 

#  of  sea rch  te r ms 

'  s 

26 

total 

#  of  commands 

s 

29 

total 

#  of  cycles 

. 

5 

bn-1 ine  connect  time 

0.497 

of  f  preparation  time 

. 

D.250 

total 

searching  time 

s 

0.747 
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Searcher  No.  005 


QUEST-DBASE 
NUMBER 

bdidii 


TOTAL  # 
RELV 

27 


TOTAL  # 
PREL 

46 


TOTAL  # 
NREL 

75 


TOTAL  # 
EVAL 

148 


tOtAL  # 
NEVL 

626 


tbtAL  # 
REtRiEVED 

774 


#  of  relevant,  items 

#  of  partially  rely 

#  of     not  releyaht 


7 
2 
O 


total  #  eyaiuated 
#     hot  evaluated 


9 
11 


to'bal  #  retrieved 


reca 1 1 
precision 


:  0.123 
:  1.000 


total  #  of  search  terms:  12 
total  #  of  commands  :  22 
total  #  of  cycles  :  6 

on-line  cdhnect  time  :  D.370 
off   preparation  time       :  0.117 


total  searching  time 


0.487 


012013 


21 


121 


148 


90 


>38 


#  of  relevant  items  :  0 

#  of  partially  reJ.y  :  3 

#  of     not  relevant  :  30 

total  #  evaluated  :  33 

#  not  evaluated  :  17 

total  #  retrieved  :  BO 

recall  :  0.111 

precision  :  0.090 

total  #  of  seat*ch  terms:  19 

total  #  of  cbrwmahds  :  17 

total  #  of  cycles  :  4 

on-line  connect  time  :  0.420 

off  preparation  time  :  0.250 

total   searching  time  :  0.670 
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Searcher  No.  005 


QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  «       TOTAL  #       TOTAL  «       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVEC 


016011  25  37  108  170  O  170 


37  108 
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#  of   reievcint/  items 

2 

#  of   partial i y  rel v 

3 

#     of     not  relevant 

: 

3 

tota 1  #  eva 1 uated 

M 

8 

#     not  evaluated 

: 

0 

total  #  retrieved 

n 

8 

recal 1 

S 

d 

.080 

precision 

S 

d 

.625 

total  #  of  search  terms 

S 

13 

total  #  of  commands 

s 

11 

total  #  of  cycles 

s 

3 

on— 1 ine  connect  time 

s 

0 

.154 

of  f   prep^ ration  time 

s 

0 

.117 

total  searching  time 

s 

0 

.271 

019075 


49 


74 


150 


'2Qx 


#  of   relevant  items 

#  of   partial ly  rel v 

#  of     not  relevant 


4 
5 
3 


total  #  evaluated 
#     not  evaluated 


12 
1 


total  #  retrieved 

recal 1 
precision 

total  #  of  search  terms 
tota 1  #  o f  command s 
tota 1  #  of  eye 1 es 

on— 1 ine  connect  time 
off   preparation  time 


13 

0.118 
0.750 

9 
10 
4 

Q.189 
0.133 


tota 1  sea rch  i  ng  t i  me 


0.3: 


Searcher  No;  QQB 


QOEST-DBASE 
NUMBER 

D20015 


□27013 


TOTAL  # 
RELV 

26 


TOTAL  # 


TOTAL  # 


TOTAL  # 


35 


PREL  NREL 

EVAL 

A3  81 

150 

#  of   re  1  eyant  i'terfiB 

- 

1 

#  of   partial ly  rely 

s 

□ 

#     of     hot  relevant 

s 

3 

total  #  evaluated 

■ 

#     riot  evaluated 

■ 
■ 

S 

total  #  retrieved 

9 

recal 1 

a 

0.014 

precision 

0.25D 

total  #  of   search  terms 

*  Z 

15 

total  #  of  commands 

s 

17 

total  #  of  cycles 

s 

dh~lihe  cbnhect  time 

s 

Q.3A6 

of f  preparatibh  time 

■ 

0.133 

total  searching  time 

s 

0.A79 

69  A6 

150 

#  of  re 1 eyant  items 

. 

7 

#  of   partially  relv 

s 

11 
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s 

1 

total  #  eyaliiated 

19 

#     hot  evaluated 

m 

12 

total  #  retrieved 

31 

recal i 

m 

0 . 1 7^ 

jDrecisibh 

m 

0 . 947 

total  #  of   search  terms 

m 

22 

total  #  of  commahds 

m 

14 

tbta 1  #  bf  eye 1 es 

m 

2 

bh-1 ihe  connect  time 
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of  f   preparatibh  time 

m 

m 
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total   searching  time 

m 
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TOTAL  # 
NEVL 
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TOTAL  # 
RETRIEVEC 
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58 
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Searcher  No.  006 


QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVEC 

002218  37  36  156  229  C3  229 


36  156 
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4 
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r 
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2B 
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s 

Q 
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- 

25 
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■ 
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4 
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s 

3 
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s 
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s 
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87 
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1 

TT  OX    paruiaiiy  reiv 

■ 

w 
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2 
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M 

0 

total  #  retrieved 
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6 
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precision 
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total  #  of  search  terms:  6 
total  #  of  commands  :  ii 
tota 1  #  Df  eye 1 es  :  2 

on-l ine  connect  time  :  0  s 234 
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Searcher  No.  006 
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TOTAL  # 
RELV 


TOTAL  # 
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TOTAL  # 
NREL 


TOTAL  # 
EVAL 


TOTAL  # 
NEVL 
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RETRIEVED 
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#     of     not  rei  evantr 

s 

d 
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u 
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•1 

total  #  retrieved 
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■ 

1 
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■ 

0 

.odd 
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6 
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■ 

4 
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s 

7 
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m 

3 
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0 
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m 
m 

0 
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336 
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total  searching  time 
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2 
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s 

s 
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s 

19 
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s 

29 
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s 

1 
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s 

3d 
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precision 

s 
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6 
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ii 
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i 
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s 
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0 

.250 
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s 
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Searcher  No •  006 


QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #  TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER               RELV             PREL             NREL  EVAL  NEVL  RETRIEVE 

014151  35  71  51  157  2  159 

#  of   relevant  items  :  ii 

#  of   partially  reiy  :  19 

#  of     not  re 1 evant  :  7 
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#  evaluated 

s 

37 

#     not  evaluated 

s 

2 
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s 

39 
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■ 
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s 

0.811 
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s 

12 
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s 

ii 
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s 

2 
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s 
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s 
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s 
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IS 
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s 

ii 
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■• 

id 
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m 

39 
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m 

30 
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S 

69 
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. 
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total  #  of   search  terms 

S 

6 
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M 

. 

15 
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S 

2 
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S 
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0.i67 

tota 1  search ing  ti me 
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334 


Seardher  No.  D07 


QUEST-DBASE       TOTAL  #       TOTf^L  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  S 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVE£ 
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35 
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a 

1 
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a 
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a 

7 
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u 
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a 
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a 

5 
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of  f   preparation  time 

a 
a 

0a200 

tot a 1    search  inn  tim^ 

a 

-  -    -  - 
0  a  489 

23  39 

67 

#  Df   re 1 evant  i terns 

1 
a 

4 

#  of   partially  relv 

a 
a 

14 

#     of     not  relevant 

a 
a 

18 

total  #  evaluated 

a 



36 
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36 
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a 
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a 
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a 

9 
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a 
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a 

4 
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a 
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a 

0.200 
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a 
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Searcher  No.  007 


oiUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRiEVEl 

630071  57  25  13  95  0  95 
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s 

ID 
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s 

0 
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ID 
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■ 

0.097 
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■ 
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s 

10 
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m 

14 
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m 

4 
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m 
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m 

0.066 
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m 
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s 
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19 
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s 

20 
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s 

0 
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s 

20 
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s 
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s 
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total 
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s 

13 
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s 

11 
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s 

3 

on- 1  i  rie  cbririec t  t  i  ifne 

s 
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s 
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s 
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QUESt-DBASE 
NbHBER 

036090 


Searcher  Noi  Qd7 

tOTAb  #       T©TAb  #       tOTAb  #  TQtAb  # 
REbV             PREb            NREb  EVAb 

62  49  39  150 

#  of  relevant  items  :  13 

#  of .partially  rely  :  13 

#  of     riot  relevant  :  11 

total  #  evaluated  :  37 

#  riot  evaluated  :  36 

total  #  retrieved  :  73 

recall  :  0.234 

j3recisibn  :  0.703 

total  #  of  search  terms:  13 

total  #  of  commands  :  20 

total  #  of  cycles  :  7 

dri-line  corinect  time  :  0.341 

off   j3rei3aratidri  time  :  0.167 

total  searching  time  :  0.508 


TOTAb  # 
NEVb 

57 


TOTAb  # 
RETRIEVED 

207 


ERIC 
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Searcher  No.  008 


QUEST-DBASE 
NUMBER 

019075 


TOTAL  » 
RELV 

27 


TOTAL  *f 
PREL 

49 


TOTAL  # 
NREL 

74 


TOTAL  # 
EVAL 
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TOTAL  # 
NEVL 

52 


TOTAL  # 
RETRIEVED 
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#  of   relevant  items 

m 

6 

#  of   partial ly  relv 

s 

15 

#     of     not  relevant 

s 

15 
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s 

36 

#    not  evaluated 

s 

4 

total  #  retrieved 

s 

40 

recal 1 

s 
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m 
m 
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total  #  of  search  terms 

m 

8 

total  #  of  comitisnds 

S 

26 

total  #  of  cycles 

S 

9 

on-line  connect  time 

S 
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of  f  preparation  time 

S 
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total  searching  time       :  0-405 
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0 
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83 
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4 
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total  searching  time  : 
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Searcher  Noi  QdS 


QUESt-bBASE 
NUMBER 

032CD08 


TOTAL  # 
RELV 

113 


TOTAt  # 
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19 


tOTAt  # 
NREb 

18 


tOtAb  # 
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74 
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total  #  of  cycles  : 

bri-line  connect  time  : 
off  preparation  time  : 


total  searching  time 
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12 

0.633 
0  i  333 


0i966 
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44 


57 
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#  of  partially  relv 

#  of     riot  relevarit 
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0 
3 
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2 
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reca 1 1  : 
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total  #  of  commands  : 
total  #  of  cycles  : 


bri-lirie  cciririect  time 
off  preparation  time 
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16 
4 

G.450 
0.250 


total  searching  time 
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ERIC 
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Searcher  No.  008 


QUEST-DBASE 
NUMBER 

034013 


TOTAL  # 
RELV 

10 


TOTAL  # 
PREL 

39 


TOTAL  # 
NREL 

100 


TOTAL  # 
EVAL 
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TOTAL  # 
NEVL 
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TOTAL  » 
RETRIEVEI 
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12 
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s 
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m 
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m 
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8 
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m 

6 
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m 
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m 
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■ 
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65 
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6 
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5 
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S 

5 
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16 
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s 

0 
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■ 

16 
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m 
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S 
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total  #  of  search  terms 

S 

20 
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m 
m 

22 
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m 

6 

on-i ine  connect  time 

m 
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=\ 
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total  searching  time 

s 
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Searcher  No;  009 


QUEST-DBASE 
NUMBER 
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tdtAb  # 
RELV 

60 


TOTAt  # 
PREL 

58 


TOTAt  # 
NREL 

33 
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EVAL 
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TOTAL  # 
NEVL 

8 
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RETRIEVED 
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11 
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6 
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8 
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0 
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3 
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Searcher  No.  009 
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010154 
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1 
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Searcher  No.  1309 


QOEST-DBASE       TOTAL  tt       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  4*       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRiEVEI 

015154  28  86  36  150  172  322 
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-total  searching  time 
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61 


0 


61 


#  of   relevant  items 

#  of   partially  relv 

#  of     not  relevant 


2 
51 


total  #  eva 1 aated 
#     not  evaluated 


56 
Q 


total  #  retrieved 


56 


recal 1 
precision 


:  0.714 
:  0.089 


total  #  of  search  terms; 
total  #  of  commands  : 
total  #  of  cycles  : 


total  searching  time 


8 


bn-lirie  connect  time  :  0.156 
off  preparation  time       :  0i233 


:   B.i  9 


o 

ERIC 
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Searcher  No;  010 


QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  j=>REL  NREL  EVAL  NEVL  RETRIEVED 

010154  4  IB  130  149  310  459 


15  13Q 

149 

#  of  relevant  items 

s 

0 

#  of  partially  relv 

s 

b 

#     of     not  relevant 

s 

28 

total  #  evaluated 

s 

28 

#     not  evaluated 

• 

72 

tota  i  #  retrieved 

s 

iob 

recal 1 

s 

d.QQd 

precision 

s 

0.000 

total  #  of  search  terms: 

7 

total  #  of  commands 

s 

4  5 

total  #  of  cycles 

g 

on-line  connect  time 

« 

0.234 

off   preparation  time 

s 

0.200 

tota 1   searching  time 

s 

0.434 

25  115 

149 

#  of  relevant  items 

s 

3 

#  of  partially  relv 

6 

#     of     not  rei evant 

31 

total  #  evaluated 

s 

40 

#     not  evaluated 

■ 

58 

total   #  retrieved 

s 

98 

recal 1 

s 

0.265 

precision 

s 

0 . 225 

total   #  of  search  terms 

m 
m 

9 

total  #  of  commands 

m 

5 

total  #  of  cycles 

m 

2 

01115A  9  25  115  149  161  310 


9P~^  ?r^?_99^r^®5^  ^i'"?  •  Qil7Q 
of f   preparation  time       :   □ . 250 

total  searching  time       :  0.420 
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Searcher  Na.  OiQ 


QUEST-DBASE 
NUMBER 


TOTAL.  # 
RELV 

28 


TOTAL  # 
PREL 

86 


TOTAL  # 
NREL 

36 


TOTAL  # 
EVAL 

150 


TOTAL  # 
NEVL 

172 


TOTAL  # 
RETRIEVED 

322 


#  pf   relevant  items         :  27 

#  of   partial ly  rely         :  61 

#  of     not  re ley ant         s  30 


total  #  evaluated 
#     hot  evaluated 


118 
144 


total  #  retrieved  s  262 

recall  :  0.772 

precision  :  0.746 

total  #  of  search  terms:  1 

total  #  of  commands  :  6 

total  #  of  cycles  :  1 

ori-lirie  connect  time  :  0.1 91 

off   preparation  time  :  0.133 


total   searching  time       :  0.324 
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Searcher  No.  Oil 


QUESTrDBASE 
NUMBER 


006006 


TOTAl.  # 
RELV 

11 


TOTAt  # 
PREt 


TOTAt  # 
NREL 


EV^'. 


^  ISA 

■ 

0 

#  of   partially  relv 

s 

b 

#     of     not  relevant 

■ 
■ 

□ 

- 

□ 

#     not  evaluated 

s 

1 

tot;^  1   4t  retrieved 

m 

1 

recai 1 

M 
■ 

0.000 

precision 

S 

0.000 

total  #  of  search  terms 

s 

12 

total  #  of  commands 

■ 

41 

total  #  of  cycles 

s 

6 

ori-1 ine  connect  time 

1  .123 

off   preparation  time 

s 

0.833 

TOTAL  % 
NEVL 

8S 


TOTAL  # 
RETRIEVED 

238 


total   searching  time 


1 .  9B6 


012013 


121 


148 


90 


238 


#  of   relevant  items 

#  of   partially  relv 

#  of     hot  relevant 


4 
9 
31 


total  #  evaluated 
#     hot  evaluated 


44 
38 


total 

#  retrieved 

• 
■ 

82 

recal 1 

■ 

6.481 

precision 

■ 

■ 

0 . 295 

total 

#  of  search  terms 

s 

33 

total 

#  of  commands 

s 

37 

total 

#  of  cycles 

2 

bn-1 irie  connect  time 

s 

6.898 

off   preparation  time 

s 

0.2*50 

total 

searching  time 

■ 

1 .148 
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Searcher  No.  Oil 


QUEST-DBASE 
NUMBER 

017005 


TOTAL  # 
RELV 

36 


TOTAL  # 
PREL 


TOTAL  « 
NREL 

sa 


TOTAL  # 
EVAL 

150 


TOTAL 
NEVL 

377 


TOTAL  # 
RETRIEVED 

527 


#  of 

relevant  items 

6 

#  of 

partial iy  reiv 

• 

9 

#  of 

not  relevant 

38 

total 

#  evaluated 

B 

a 

53 

#     not  evaluated 

m 
m 

64 

total 

#  retrieved 

S 

117 

recal 1 

S 

0-242 

precision 

B 

■ 

0.283 

total 

#  of  search  terms 

S 

11 

total 

#  of  commands 

m 
m 

20 

total 

#  of  cycles 

m 

2 

dn-1 ine  connect  time 

S 

□  -  378 

of  f   preparation  time 

m 
m 

0.250 

total 

searching  time 

m 
m 

0.628 

69  46 

150 

#  of  relevant  items 

s 

7 

#  of   partially  relv 

s 

19 

#     of     not  relevant 

s 

13 

total  #  evaluated 

s 

39 

#     not  evaluated 

s 

15 

total  #  retrieved 

s 

54 

recal 1 

M 

. 

0.250 

precision 

0.667 

total  #  of  search  terms 

■ 

. 

11 

total  #  of  commands 

s 

39 

totai  #  of  cycles 

s 

7 

on-l ine  connect  time 

s 

0 . 499 

of  f   preparation  time 

s 

0.250 

total   searching  time       :   0 . 749 
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Searcher  No.  Oil 

QUEST-DBASE       TOTAL  #       TOTAL  #  TOTAL  #  TOTAL  #  TOTAL  #  TOTAL  # 

NUMBER               RELV             PREL             NREL  EVAL  NEVL  iRElRIEVEC 

b29CDb8                   36                 34                 SO  150  77  227 


«  of 

re  i  evant  items 

s 

& 

#  of 

partial ly  relv 

s 

& 

#  of 

not  relevant 

s 

total 

#  evaluated 

s 

28 

#     not  evaiaated 

s 

22 

tota  1 

#  retrieved 

■ 
■ 

50 

recal 1 

s 

d.229 

precision 

s 

0.571 

total 

#  of  search  terms 

s 

24 

total 

#  of  commands 

32 

total 

#  of  cycles 

s 

2 

on-1 ine  connect  time 

s 

0.076 

of  f  preparation  time 

• 

0.683 

tota  i 

search ing  time 

m 

0.159 
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Searcher  Na.  012 


QUEST-DBASE 
NUMBER 

024191 


TOTAL  # 
RELV 


TOTAt  # 
PREt 


TOTAt  # 
NREL 

30 


TeTAb  # 

EVAt 
35 


TOTAL  # 
NEVL 


TOTAb  # 
RETRIEVED 

35 


#  of   relevant,  items 

#  of   partially  relv 

#  of     riot  relevarit 


1 
1 
4 


total  #  evalaated 
#     riot  evalaated 


6 
0 


total  #  retrieved 

recall 
precision  i 

total  #  of  search  terms; 
total  #  of  commands 
total  #  of  cycles  ; 

on— line  connect  time  ! 
off   preparatiori  time  ! 


0.400 
0 . 333 

14 
13 
2 

0.267 
0.333 


total  searchirig  time 


0.600 


39  8 

84 

#  of   relevant  items 

s 

1 

#  of  partial ly  relv 

■ 

0 

#     of     not  relevant 

s 

0 

to ta 1  #  e va 1 uated 

s 

1 

#     not  eva 1 uated 

s 

D 

total  #  retrieved 

1 

recal 1 

s 

0 

.013 

precision 

■ 

1 

.000 

total  #  of  search  terms 

s 

5 

total  #  of  commands 

■ 
m 

9 

total  #  of  cycles 

m 

3 

bn~l irie  connect  time 

m 

0 

.208 

of  f   preparation  timie 

m 

0 

.  333 

total  searching  time 

m 

c 

0 

.541 
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Searcher  No.  012 


QUEST-DBASE 
NUMBER 


026038 


TOTAL  # 
RELV 


TOTAL  # 
PREL 

23 


TOTAL  # 
NREL 

39 


TOTAL  # 
EVAL 

67 


TOTAL  # 
NEVL 


TOTAL  # 
RETRIEVEC 

67 


#  of  relevant  items 

#  of  partially  relv 

#  of     not  relevant 


4 

5 


total  #  evaluated 
#    not  evaluated 


11 
0 


total  #  retrieved 


11 


reca 11  :  0 . 321 

precision  :  0.818 

total  #  o£  search  terms:  8 

total  #  of  commands  :  9 

total  #  of  cycles  !  i 

on-line  connect  time  :  0.150 

off  preparation  time  :  0.250 


total  searching  time 


0.400 


030071 


57 


13 


9% 


95 


#  of   relevant  items         :  29 

#  of   partially  relv         :  14 

#  of     not  relevant  :  S 


total  #  evaluated  :  48 

#     not  evaluated  s  Q 


total  #  retrieved  :  48 

recall  :  £ji524 

precision  :  0.896 

total  #  of  search  terms:  1(3 

total  #  of  commands  :  12 

total  #  of  cycles  :  i 

on-line  connect  time  :  0^183 

off  preparation  time  :  0.250 


total  searching  time       :  0.433 
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Searcher  Nci  012 


QUEST-DBASE 
NUMBER 


031061 


036090 


TOTAL  # 
RELV 

14 


TOTAL  # 
PREL 


TOTAL  # 
NREL 


TOTAL  # 
EVAL 


62 


15  85 

114 

#  of  relevant  items 

s 

6 

#  of  partially  relv 

s 

6 

#     of     not  relevant 

34 

total  #  evaluated 

s 

46 

^     not  eva  1  LiatecJ 

■ 

0 

total  #  retrieved 

5 

46 

recal 1 

■ 
* 

0.414 

precision 

S 

0-261 

total  #  of  search  terms 

>  S 

4 

total  #  of  commands 

■ 
* 

10 

tota 1  #  of  eye 1 es 

S 

3 

on""line  connect  time 

0.141 

of  f   preparation  time 

S 

0.167 

0 . 30S 

49  39 

150 

#  of   relevant  iteftis 

s 

5 

#  of   partially  lelv 

s 

2 

#     of     not  re 1 evant 

z 

5 

total  #  ev^luatei^ 

m 
m 

12 

#     not  eval .  ,^ted 

m 

0 

total  #  retr; sved 

m 

12 

reca? 

m 

0.063 

precisicT: 

m 

TOTAL  # 
NEVL 


TOTAL  # 
RETRIEVED 

114 


57 


207 


total  #  of  soar  .h  tc.- - 
total  #  of  •c:,-.irfi  :-Kic 
total  #  of  c;yi:;lc;'-5 


li« 


on— 1 ine  connect  t  ^  r 
of  f   preparatir>n  t  >. «:*  . 

tota  1  sea r ch  i  n^  1 1  it>e 


5    O   :  JO 
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Searcher  No.  013 

QiJESt-bBASE       TOTAL  #       TOTAh  #       TOTAb  #       ToTAb  #       TOTAL  #       TOTAb  # 

NUMBER               REtV             PREt             NREt  EVAb             NEVb  RETRIEVED 

Obi  Oil                   27                 46                 75  448  626  774 

#  of  relevant  itemB  :  5 

#  of  partially  relv  :  3 

#  of     not  relevant  s  D 


tota 1  #  eva 1 uated  :  8 

«     not  evaluated  :  28 

total  #  retrieved  :  36 

recall  :  0.110 

precision  r  liOOO 

total  #  of  search  terms:  13 

total  #  of  commands  :  11 

tbta  1  #  C3f  eye  1  es  :  3 

on-1 ine  connect  time  :   0 , 301 

off   preparation  time  :  0.333 

total  searching  time  s   0 , 634 


0051  AS  16  23  48  87  1  88 

#  of  relevant  items  :  0 

#  of  partially  relv  :  0 

#  of     hot  relevant  :  0 


total  #  evaluated  :  0 

#     not  evaluated  :  1 

total  #  retrieved  :  1 

recall  :  0.000 

preoision  :   0. 000 

total  #  of  search  terms:  21 

total  #  of  commands  :  13 

total  #  of  cycles  :  3 

on- 1 ihe  connect  t i me  :   0 . 345 

off  preparation  time  :  0,167 

total   searching  time  :  0.512 
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Searcher  No.  013 


QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVEE 

013015  7  36  106  149  14  163 


«  of 

relevant  items 

s 

3 

»  of 

partial ly  rel v 

s 

18 

#  of 

riot  relevarit 

s 

39 

t;ot;dl 

#  evaluated 

s 

60 

#     not  evaluated 

■ 
■ 

5 

total 

#  retrieved 

s 

65 

recal 1 

■ 
■ 

0.48S 

precision 

■ 

m 

0.350 

total 

#  of  search  terms 

s 

6 

total 

#  of  commands 

■ 

11 

total 

#  of  cycles 

s 

3 

bri- 1  i  rie  connect  t i  me 

■ 

0.206 

of  f   preparation  time 

s 

0.0S3 

tota 1   search  i       t i  me       :   0 . 287 


37  108 

170 

#  of   relevant  items 

s 

4 

#  of   partial ly  rel v 

s 

4 

#     of     riot  relevarit 

s 

37 

tbta 1  #  eva 1 uated 

s 

45 

#     not  evaluated 

s 

0 

total   #  retrieved 

s 

45 

recal 1 

s 

6.129 

:  precision 

■ 

0.178 

total   #  of  search  terms 

a 

2 

total   #  of  commands 

s 

9 

total  #  of  cycles 

s 

3 

bn-1 irie  connect  time 

■ 

0.141 

bf  f   preparabibri  time 

s 

0.117 

total   searching  time       s  0.258 


Searcher  No.  013 


QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVED 

019075  27  49  74  150  52  202 

#  of  relevant  items  :  6 

#  of   partial ly  relv  :  12 

#  of     not  re 1 e vant  :  2 

tota 1  #  e va 1 Uated  :  20 

#  not  evaluated  :  5 

total  #  retrieved  5  25 

recall  :  0.237 

precision  :  0.900 

total  #  of  search  terms:  5 

tota 1  #  of  commands  :  8 

total  #  of  cycles  :  1 

on— line  connect  time  s  0.157 

off   preparation  time  :  0.167 

total  searching  time  :  0.324 


020015  26  43  81  150  161  311 


#  of 

re 1 evant  items 

m 
• 

5 

#  of 

partial ly  rely 

11 

#  of 

hot  r  ..levant 

m 
• 

33 

total 

#  evaluated 

* 

^9 

#     not  evalLi^ated 

m 
• 

^3 

total 

#  retrieved 

• 

92 

recal 1 

• 

0.232 

precision 

• 

0.327 

total 

#  of  search  terms 

m 
• 

5 

total 

#  of  commands 

m 
• 

total 

#  of  cycles 

m 
• 

A 

bn-1 ine  connect  time 

m 
• 

0.141 

off   preparation  time 

• 

0.117 

total  searching  time       s  0.258 
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Searcher  No.  014 


QUEST-DBASE 
NUMBER 

003064 


TOTAL  # 
RELV 

36 


TOTAL.  # 
PREL 

47 


TOTAL  # 
KiREt 

68 


TOTAL  # 
EVAL 

iSi 


Total  # 

NEVL 
272 


TOTAL  # 

retrieved' 


423 


#  of   relevant  items         :  i 

#  of  partially  relv         '  ^' 

#  of     not  relevant 

total  #  evaluated 

#  not  evaluated 


total  #  retrieved 


16 


recal 1  

precision 


:   0 . 036 

:  Or75G 


total  #  of  search  terms:  6 
total  #  of  commands  :  4 
total  #  of  cycles  s  1 

on-line  connect  time       :  0iQS9 

cff  preparation  time       :  C3.i67 


total  searching  time 


d.: 


007075 


70 


39 


40 


149 


335 


484 


#  of   relfjvaht  items         :  1 

#  of   partially  relv         :  0 

#  of     not  relevant         :  1 


total  #  evaluated 
#     not  evaluated 


10 


total  #  retrieved 


i: 


recall  :  0.009 

precision  :  0.500 

tctai  #  of   search  terms:  9 

total  #  of  coff.jnahds  :  6 

total  #  of  cycles  :  1 

on-line  connect  time  s  0^063 

off  preparation  time  :  0^117 


total  searching  time 


O.lSO 
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Searcher  No.  014 


QUESt-bBASE 
NUMBER 

009£337 


tdtAt  # 
RELV 

18 


TOTAL  # 
i=»REt 

48 


TOTAL  # 
NREL 

84 


TOTAL  # 
EVAt 

15Q 


TOTAL  # 
NEVL 

495 


TOTAL  # 
REtRiEVEb 

645 


#  of   relevant,  items 

#  of   fjartially  relv 

#  of     hot  relevaht 


29 


tot a 1  #  eva 1 aa ted 
#     not  evalaated 


37 
97 


total  S  retrieved 


:  134 


recaii  i 

jarecision  i 

total  #  of  search  terms: 

total  #  of  commands  s 

total  #  of  cycles  s 

on-line  connect  time  : 

off   preparation  time  : 


0.121 
0.216 

3 
8 

2 

O.lli 
d.25d 


total  searching  iime 


0.361 


018015 


66 


38 


46 


150 


61 : 


76i 


#  of   relevant  items 

#  of   partially  relv 

#  of     hot  relevaht 


O 
O 
O 


total  #  evaluated 
#     hot  evaluated 


O 
O 


total  #  retrieved 


O 


recall  :  O.OOO 

jDrecision  :  0.000 

total   #  of   search  terms:  5 

total  #  of  commands  s  5 

total  #  of  cycles  :  2 

on-line  connect  time  :  G,193 

off   preparation  time  s  d.085 


total   searching  time 


0 . 276 
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QUEST- DBASE       TOTAL.  #       TOTAL  #       TOTAL  #       TOTAL  #       tOTAL  #       tOtAt  # 
NUMBER  RELV  PREL  NREL  lEVAL  KlEVL  REtRIEVED 

021037  19  6  77  lQ2  O  102 


#  of 

relevant  items 

m 
m 

1 

#  of 

partial ly  relv 

m 
m 

0 

S  of 

not  relevant 

S 

0 

'total 

#  evaluated 

S 

1 

#     not  evaluated 

m 
m 

0 

total 

#  retrieved 

m 

1 

recal 1 

0 

.040 

precision 

m 

1 

.000 

total 

#  of   search  t5?rms 

S 

4 

total 

#  of  comma  rios 

S 

11 

total 

#  of  cycles 

3 

■2, 

on- i  i ne  connect  tifiie 

m 

0 

.230 

off   preparation  time 

M 

■ 

0 

.133 

total 

searching  ti  me 

S 

0 

.  363 

025001  30  26  94  150  430  580 


26  94 

150 

#  of   re 1 evant  items 

m 
m 

3 

#  of  partially  relv 

m 

2 

#     of     not  relevant 

m 
m 

6 

total  #  evaluated 

m 

11 

#     not  evaluated 

m 

4S 

total  #  retrieved 

m 

59 

recal 1 

m 

0.089 

precision 

m 

.0.455 

total  #  bf  search  terms 

S 

5 

total  #  of  com m a nd s 

m 
m 

6 

tota 1  #  of  eye 1 es 

m 

2 

on-1 ine  connect  time 

m 

0.094 

bf  f   prejDaratibh  time 

m 
m 

0.117 

total  searching  time 

m 

0.211 
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Searcher  No.   01 S 


QUEST-DBASE       tOtAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVED 

023032  29  9  51  89  O  89 

#  of   relevant  items  s  23 

#  of  partially  relv  :  3 
^     of     not  relevant  s  9 


total  #  evaluated  :  35 

#     not  evaluated  :  D 

total  #  retrieved  :  35 

recall  :  0.684 

precision  :  0,743 

tota 1  #  of   search  terms  s  8 

total  #  of  commands  :  22 

total  #  of  cycles  :  3 

on-line  connect  time  5  0.487 

off   preparation  timc^  :  0.5D0 

total   searching  time  s  0.987 


026038  37  39  8  84  0  84 


#  of  relevant  items         :  □ 

#  of   partial ly  relv         :  0 

#  of     not  relevant         :  3 


tota 1  #  eva 1 uated 

. 

3 

#     not  evaluated 

s 

D 

total   #  retrieved 

s 

3 

reca 1 1 

s 

o.ooo 

precision 

s 

O.DOO 

total  #  of  sejirch  terms 

s 

•15 

total   #  of  commands 

s 

•1? 

total  #  of  cycles 

. 

3 

on— 1 ine  connect  time 

m 
m 

Q .  255 

of  f   preparation  time 

m 

0.250 

tota 1  search ing  time 

S 

6.505 
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QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  »       TOTAL  » 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVED 

037016  78  18  BA  150  69  219 


#  of   relevant  items 

• 

m 

11 

#  of   partial ly  rel v 

m 

3 

#     of     not  re 1 evant 

m 

total  #  evaluated 

m 

^7 

#     not  eval uated 

m 

25 

total  #  retrieved 

m 

72 

recal 1 

m 

0.146 

jprecisibn 

m 

0.298 

total  ft  of  search  terms 

m 

7 

tbta  i       of  commands 

m 

26 

tota  I  #  c.:   eye  1  es 

m 

5 

bn-1  ine  co  ..lect  time 

m 

0.  733 

off   preparation  time 

S 

0.500 

total  searching  time 

5 

1  . 233 

038061 


7'^ 


150 


595 


745 


»  of   relevant  items 

#  of   jaartially  relv 

#  of     not  relevant 


8 
5 
3 


total  #  evaluated 

s 

16 

#     not  evaluated 

53 

total   #  retrieved 

m 

69 

recall 

m 
m 

0.120 

precision 

m 
m 

b;Si3 

total  #  of  search  terms 

m 
m 

16 

total  #  of  commands 

S 
m 

24 

total  #  of  cycles 

m 
m 

3 

on-1 ine  connect  time 

m 

m 

0.361 

of  f   preparation  time 

m 

m 

0.5QQ 

tota 1  search ing  time       :  O • 861 


360 


Searcher  No.  015 


QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVE! 

039015  26  38  102  166  O  166 


#  of  relevant  items 

s 

0 

#  Df  partial ly  relv 

s 

0 

#     of     not  re 1 evant 

s 

(J 

total  #  evaluated 

■ 

0 

#     not  evaluated 

■ 
■ 

0 

total  #  retrieved 

■ 

0 

recal 1 

s 

0 . 000 

precision 

■ 

0.000 

total  #  of  search  terms 

s 

10 

tota 1  #  of  commands 

a 

10 

total  #  of  cycles 

■ 
m 

3 

on-1 ine  connect  time 

m 

0.166 

off  prepare!: ion  time 

m 

0.500 

tota 1  search i ng  t i me 

u 

0.666 

040016  77  40  32  149  331  480 

#  of  relevant  items  :  10 

#  of  partial iy  relv  :  6 

#  of     not  relevant  :  O 


total  #  evaluated  :  16 


#     not  evaluated 

s 

61 

total 

#  retrieved 

s 

77 

recal 1 

s 

0.137 

precision 

s 

1  . 000 

total 

#  of  search  terms 

s 

8 

total 

#  of  commands 

s 

32 

total 

#  of  cycles 

s 

9 

on-1 ine  connect  time 

0.443 

off  preparation  time 

s 

0.250 

total 

searching  time 

s 

0.693 

361 


Searcher  No.  016 


QUEST-DBASE 
NUMBER 


TOTAL  » 
RELV 


TOTAL  # 
PREL 


TOTAL  # 
NREL 


TOTAL  # 
EVAL 


TOTAL  # 
NEVL 


TbtAL  # 
RETRIEVED 


001011 


27 


003064 


36 


Ha                           r  Z> 

- 
s 

■i 

X 

#  of   partially  reiv 

9 

#     of     not  re 1 evant 

■ 

25 

- 
■ 

#     not  evalaated 

■ 
■ 

d 

T/Oua  1  tF  rex/r xe veo 

- 

s 

OS 

recal 1 

■ 
■ 

0.137 

precx  5xon 

s 

Lf  ■  ^OO 

tota  i  #  of  search  terms 

'  s 

9 

total  #  of  commands 

s 

total  #  of  cycles 

e 

1 

on-line  connect  time 

: 

G.2QS 

of f   preparation  time 

s 

0 . 333 

tota \  searching  time 

s 

0.541 

X  13  1 

tt  oi   reievanx/  xuems 

- 
■ 

n 

w 

#  of   partial ly  rel v 

s 

#     of     not  re 1 evant 

s 

X/OX/a  X  ?t  eva  x  ciax/ea 

- 

■ 

n 

w 

#     not  evalaated 

s 

117 

  _._  

tota 1  #  retrieved 

_ 
■ 

4-17 
X  1  r 

recal 1 

• 
■ 

0.000 

precision 

■ 

0.000 

total  #  of   search  terms 

7 

total  #  of  commands 

m 
m 

17 

tot a 1  #  of  eye 1 es 

m 

1 

on-1 ine  connect  time 

0.536 

off   preparation  time 

m 

0 . 333 

total  searching  time 

m 

0.869 

626 


774 


272 


423 


362 


QUEST-DBASE 
NUMBER 

009037 


016011 


TOTAt  # 
RELV 

18 


Searcher  No.  016 
tOTAt  #       TbTAt  # 


TOTAt  # 


PREt  NREL 

EVAt. 

48  84 

150 

#  of  re  1  evant  i terns 

■ 

m 

d 

#  of  fDartially  relv 

S 

#     of     not  relevant 

S  < 

.  27 

total  #  evaluated 

S 

30 

#     not  evaluated 

m 

54 

t^tal  S  retrieved 

m 

84 

recal 1 

m 

0.045 

precision 

m 

OilOO 

tota 1  #  of  search  terms 5 

6 

total  #  of  commands 

m 
m 

l  3 

total  #  of  cycles 

m 

1 

bn-1 ine  connect  time 

S 

0.163 

of  f   preparation  time 

S 

0.500 

total   searching  time 

S 

0.663 

37  108 

170 

#  of   relevant  items 

S 

15 

#  of  partial ly  relv 

S 

17 

#     of     riot  relevant 

« 

27 

total  #  evaluated 

S 

59 

#     not  evaluated 

S 

D 

total  #  retrieved 

S 

59 

recal 1 

m 
m 

0.516 

precision 

m 

0.542 

total  #  of   search  terms 

m 

5 

total  #  of  commands 

S 

8 

tot       #  of  cycl  es 

S 

1 

on-1 ine  connect  time 

• 

0 . 1 91 

of  f  preparation  time 

S 

0.250 

total   searching  time 

■ 
• 

0.441 

TOTAt.  # 
NEVL 

495 


TOTAL  # 
RETRiEVEb 

645 


170 


363 


Searcher  No-  016 


QUEST-DBASE 
NUMBER 

021037 


TOTAL  # 
RELV 

19 


TdTAt  # 
PREt 


TOTAL  # 
NREL 

77 


TOTAL  # 
EVAL 

102 


TOTAL  # 
NEVL 


TOTAL  » 
RETRIEVEb 

102 


#  of  relevant  items 

#  of  partially  relv 

#  of     hot  relevant 


total  #  evaluated 
#     riot  evaluated 


0 

2 
28 


30 
0 


025001 


30 


#  retrieved  : 

30 

recall  : 
precision  : 

6 . 080 
0.066 

■total 
total 

#  of  search  terms : 

#  of  commands  : 

ii 

17 

total 

#  of  cycles  : 

3 

on-line  connect  trime  : 
off   preparation  time  : 

0^416 
0,500 

to-tal 

searching  time  : 

0.916 

26  94 

150 

430 


S80 


#  of  relevant  items  :  0 

#  of  partially  relv  :  0 

#  of     not  relevant  s  3 

tota 1  #  eva 1 ua ted  :  3 

#  not  evaluated  :  33 

total  #  retrieved  :  36 

recall  :  D.DDD 

precision  :   0 .000 

total  #  of  search  terms:  12 

total  #  of  commands  :  17 

total  #  of  cycles  :  2 

bn-1 irie  connect  time  :   0 .511 

off   preparation  time  :   0 . 750 

total  searching  time  :   1 .261 


ERLC 


Siearcher  No.  017 


QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVED 

019075  27  49  74  150  52  202 


*tV                           i  H 

TT    w  X     L^cr  •!  cr  V  d  1  1  w    J.  wtriii=:> 

m 
m 

7 

#  of   partrially  relv 

S 

14 

#     of     not  re 1 evant 

S 

22 

X/O ua  1   Tt  evaiuiauea 

5 

#     not  evaluated 

m 
m 

18 

__       _  _ 
tota 1  #  retrieved 

S 

61 

recal 1 

S 

0.276 

precision 

■ 

0.488 

total  #  of  search  terms 

■ 
■ 

7 

total  #  of  commands 

11 

total  #  of  cycles 

m 
m 

3 

on- 1 ine  connect  time 

S 

0.176 

aft   preparation  time 

m 

0.250 

total  searching  time 

S 

0.426 

020015 


43 


81 


ISO 


161 


311 


#  of  relevant  items 

#  of   partially  relv 

#  of     not  relevant 


O 
4 
10 


tota 1  #  eva 1 uated 

m 

14 

#     not  evaluated 

■ 

25 

total   #  retrieved 

S 

39 

recal 1 

■ 

H 

0 . 058 

precision 

S 

0 . 286 

tota 1  #  of  sea r ch  te v 

fns: 

8 

total  #  of  commands 

m 
m 

IS 

total  #  of  cycles 

m 
m 

5 

on-l ine  connect  time 

S 

0 . 227 

of  f   preparation  time 

S 

0.167 

total  searching  time 

S 

0.394 

365 


Searcher  No.  017 


QUEST-DBASE       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  #       10 1.1.   V.       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVEC 

032008  lis  19  18  150  298 


#  Df  relevant  items 

s 

□ 

#  of   partial ly  relv 

s 

1 

#     of     not  re 1 evant 

17 

total  #  evtiivtated 

s 

IS 

#     not  evaluated 

s 

□ 

total  #  retrieved 

r 

B 

IS 

recal 1 

s 

□ 

.007 

precision 

■ 

□ 

.055 

total  #  of   search  terms 

s 

9 

total  #  of  commands 

s 

9 

tota 1  #  of  eye 1 es 

s 

3 

bn-1 ine  connect  time 

s 

0 

-092 

of  f  preparation  time 

s 

□ 

.167 

total  searching  time       :   □ . 259 


57  49 

150 

#  of  relevant  items 

s 

3 

#  of  partial ly  relv 

s 

0 

#     of     not  re 1 evant 

s 

C3 

total  #  evaluated 

s 

3 

#     not  evaluated 

s 

A 

total  #  retrieved 

7 

recal 1 

s 

□ 

.029 

precision 

s 

1  .000 

total  #  of   search  terms 

s 

6 

total  #  of  commands 

s 

17 

total  #  of  cycles 

■ 

4 

on-1 ine  connect  time 

m 
m 

□ 

.324 

off  preparation  time 

m 
m 

□ 

-667 

total   searching  time       :  0.991 


366 


Searcher  No.  017 

QUEST-DBASE       TOTAL  #       TOTAt.  #       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  # 

NUMBEt=?               RELV             PREL             NI=?EL  EVAL             NEVL  RE:TI=?IEVEb 

034013                   10                 39               100  149  2D7  356 

#  of   relevant  items  :  O 

#  of  partially  relv  :  4 

#  of     not  re 1 evan t  :  5 


total  #  evaluated  :  9 

#     not  evaluated  :  1 

total  #  retrieved  :  10 

recal 1  :  QiQSl 

precision  :  0^444 

total  #  of   search  terms:  6 

total  #  of  commands  :  15 

total  #  of  cycles  :  5 

on-line  connect  time  :  0-333 

off  preparation  time  :  0.333 

total  searching  time  :  0.666 


035154                   31  20                 14  65  O  65 

#  of   relevant  items  :  1 

#  of  partial ly  rel v  :  0 

#  of     riot  rel evant  :  0 


total  #  evaluated  :  1 

#     riot  evaluated  :  0 

total  #  retrieved  :  1 

recall  :  0.019 

precision  :  1.000 

total  #  of   search  terms:  3 

total  #  of  commands  :  6 

tota 1  #  of  eye 1 es  :  1 

on-line  connect  time  :  0. DSO 

of  f  preparation  time  :   0 • 083 

total  searching  time  :  0.163 


367 


Searcher  No.  021 


QUEST-DBASE       tOTAt  #       TOTAt  #      TOTAL  «       TOTAb  #       TOTAL  «      TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVED 

bOldii  27  46  7S  148  626  774 


#  of  relevant  items 

#  of  partially  rrlv 

#  of     not  relevant 


11 
30 
41 


tota^  #  eva 1 uated 
#    not  evaluated 


82 
443 


total  #  retrieved 

recall  

precision 

total  #  of  search  te>rins 
total  #  of  commands 
total  #  of  cycles 

on-line  connect  time 
off  preparation  time 


B25 

6.562 
e.580 

12 
19 
5 

D.338 
0.250 


total  searching  time 


0.588 


003064 


36 


47 


68 


151 


272 


423 


#  of  relevant  items         r  13 

#  of  partially  relv         r  29 

#  of     not  relevant         r  34 


tota 1  #  e va 1 uated 
#    not  evaluated 


76 
85 


total  #  retrieved 

reca 1 1 
precision 


r  161 

:  0.506 
:  0.553 


total  #  of  search  terms:  8 

^btal  #  of  bbmmands  :  13 
total  #  of  .cycles            :  4 

on-line  connect  time       :  0.165 

off  preparation  tifne       :  0.250 

total  searching  time       s  0.415 


Searcher  No,  P2i 


QUE8t~DBASE 
NUMBER 

009037 


016011 


TOTAL  # 
RELV 

la 


TOTAb  « 
PREb 


TOTAb  « 
NREb 


TOTAL  # 
EVAL 


25 


4B  84 

150 

--      -       -          -    -  - 

#  of  relevant  i^bems 

1 

1 

#  of  i3ar'bial ly  rely 

c 

3 

#    of    not  relevant 

m 

2 

tc^tal  #  evaluated 

z 

6 

#    not  evaluated 

i 

52 

total  #  retrieved 

c 

58 

reoali  . 

8 

O.O&D 

preciei on 

c 

0.667 

total  #  of  search  terms 

^: 

8 

total  #  of  bbfrirnande 

c 

13 

total  #  of  cycles 

c 

on-line  cbnnecst  time 

0.157 

off  preparation  time 

1 

0.200 

total  searching  time 

« 

0.357 

37  108 

170 

M       -  -  ^  V   r  -         -                J                 -    -  - 

#  of  re 1 eyant  items 

t 

3 

#  of  j3artially  rely 

m 
m 

#     of     not  relevant 

t 

20 

tota 1  #  eva 1 uated 

: 

27 

#    tibt  evaluated 

: 

d 

total  #  retrieved 

• 

27 

recal 1 

• 

6.113 

precision 

• 

0.259 

tbtal  #  bf  search  terms 

'  s 

tbtal  #  of  cbmmands 

• 

7 

total  #  of  cycles 

s 

3 

bn"^line  connect  time 

« 

0.097 

bff  prepar-^*- <  on  time 

t 

0.250 

total  B€  time 

1 

0.347 

TOTAL  « 
NEVL 

495 


TOTAL  # 
RETRIEVE! 

64r 


O 


170 


ERIC 


Searcher  Ho,  021 


QUEST-DBASl 
NUriBER 

021037 


TOTAL  # 
RELV 

19 


TOTAL  « 
PREL 


TOTAL  # 
NREL 

77 


TOTAL  • 
EVAL 

102 


TOTAL  « 
NEVL 


TOTAL  « 
REtRIEVEO 

102 


025001 


30 


#  of  ri>l#vaint  i'tems 

1 

0 

«  of  partially  rely 

s 

1 

1 

5 

total  #  evalu^tisd 

t 

6 

#     no^  c^valtiated 

1 

d 

total  «  retri(Bved 

■ 

m 

6 

reoail 

I 

0.040 

precision 

B 

0.167 

total  #  of  eearch  terms 

• 

15 

WWvBl<l      ^                WWIIIIJICRI  IW  ■V' 

21 

mm  A 

total  #  of  cycles 

S 

5 

nn— 1  trio   nQtin^r^^  tj_in@ 

b .  327 

off  preparation  time 

s 

0.563 

total  searching  time 

s 

0.910 

26  9A 

ISO 

#  of  relevant  items 

m 
m 

1 

»  of  partially  relv 

t 

b 

W        OjL         TlOw     L  C7  X XT ▼  CI  1  1  w 

0 

total  #  evaluated 

i 

#    not  evaluated 

s 

6 

total  #  retrii^ved 

s 

7 

recai  i 

m 
m 

0.017 

precision 

m 
m 

1 .000 

total  M  of  search  terms 

>■  m 

8 

total  #  of  commands 

s 

12 

totaH  #  of  cycles 

: 

3 

on-line  connect  tima 

s 

o.i7s 

off  preparation  time 

s 

0.333 

total  searching  time 

s 

0.506 

A30 


580 


ERIC 


370 


QUEST^DBASE 
NUMBER 

024191 


026038 


TOTAL 
RELV 


37 


PREL  NREL 

KVAL 

#  Df  relevant  items 

1 

0 

#  of  partially  relv 

8 

i 

#     of     not  re 1 evant 

lb 

tota 1  #  eva 1 uated 

• 
■ 

11 

#    not  evaluated 

t 

b 

total  #  retrieved 

1 

11 

reoal 1 

s 

ri  "^Fin 

precision 

8 

0.090 

total  #  of  search  terms s 

total  #  of  Dommands 

1 

12 

total  #  cycles 

« 

m 

3 

on-line  connect  time 

1 

b.334 

off  preparation  time 

• 

b.25b 

total  searching  time 

8 

0.584 

39  6 

#  of  relevant  items 

8 

#  of  partially  relv 

19 

#    Jjf     not  relevant 

b 

total  #  evaluated 

27 

#    not  evaluated 

b 

total  #  retrieved 

27 

recal i 

b.355 

precision 

1  .000 

'to'tai  #  of  eearch^erme 
tot  a?       of  oommands 
total  ^  of  cycles 

on-line  connect  time 
off  preparation  time 


total  searching  time 


11 
13 
3 

b.218 
0 . 417 


TOTAL  « 
NEVL 


TOTAL  # 
RETRIEVEI 

35 


84 


0.635 


371 


Searcher  No.  023 


QUEST-DBASE       tOtAL  «       TOtAb  «       TOTAL  «       TOTAL  #       TOTAL  #      TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVED 

026038  S  23  39  67  O  67 


#  of  relevant  i^ems 

s 

0 

#  oi  partially  rely 

s 

1 

#     of     not  relevant 

8 

total  #  evaluated 

t 

3 

#    not  evaluated 

S 

n 

total  #  retrieved 

3 

3 

reca 1 1 

S 

0.035 

precision 

S 

0.333 

total  #  of  search  terms s 

6 

total  #  of  Dommands 

s 

a 
O 

total  #  of  cycles 

s 

1 

on— line  connect  time 

m 
m 

off  preparation  time 

8 

0.250 

totf     searching  time 

8 

0.358 

1^5  13 

95 

#  of  relevant  items 

37 

#  of  partially  rely 

#     of     not  relevant 

c 

total  #  e>^aluated 

Sh 

#     hot  evaluated 

b 

total  #  retrieved 

recal 1 

b.&22 

precision 

0.911 

total  #  of  search  terms: 

6 

total  #  of  commands 

11 

total  #  of  cycles 

3 

on— 1 i  he  cchnect  t  i  me 

0.163 

off  preparation  time 

8 

0.333 

total  searching  time 

8 

0.516 

372 


Saaroher  No.  023 

QUEST-DBASE 

TOTAL 

«       TOTAL  «       TOTAL  « 

TOTAL  1 

NUIiBER 

RELV 

PREL  NREL 

031061 

14 

15  B5 

114 

#  of  partially  relv  s 

12 

#    of     not  relevant  t 

41 

total  #  evaluated  t 

63 

#    not  evaluated  t 

0 

total       reurxevsu  i 

Ow 

reoal 1         '  i 

D.759 

precision  t 

LI  m  V 

total  #  of  search  terms;: 

28 

«iS       as  1        4^            ^                  HK  Wk  :s  >n  ^4  9 

uOuai    W   oz    wOfninnnas  i 

total  #  of  cycles  s 

6 

on-line  connect  time  « 

1.12A 

off  pi  i'paration  time  s 

0,750 

total  searching  time  : 

1,874 

□36090 

62 

A9  39 

150 

#  of  relevant  items  s 

#  of  partially  rely  : 

17 

#    of     not  re 1 evant  : 

13 

total  #  evaluated  : 

50 

#     not  evaluated  : 

12 

62 

recall  : 

0,333 

p^SwXSxori  i 

Q  -.  7AD 

total  #  of  search  terms: 

12 

total  #  of  commandi^  : 

23 

total  #  of  cycles  : 

5 

on-iine  connect  time  : 

0i597 

off  preparation  time  i 

0^583 

total  searching  time  t 

1,180 

Total  # 

NEVL 


TOTAL  « 
RETRIEVED 

114 


57 


207 


3f3 


Svaroher  No.  02^ 


QUEST-DBASE 
NUMBER 

0041 5 A 


ooei54 


TOTAL  # 
RELV 

60 


TOTAL  # 


TOTAL  #       TOTAL  « 


PREL  NREL 

EVAL 

D 

56  33 

151 

«  Df 

relevant  items  i 

26 

«  of 

partially  relv  i 

8 

«  Df 

not  re 1 evant  i 

5 

total 

#  evaluated  i 

41 

#    not  evaluated  s 

1 

total 

#  retrieved  i 

42 

reoal 1  s 

0.305 

precision  i 

0.678 

total 

#  of  fiearoh  terms s 

7 

total 

#  of  opmmands  i 

16 

total 

#  of  cycles  i 

6 

on-line  ponnect  time  s 

Q.194 

off  preparation  time  s 

Urn  lor 

total 

searching  time  s 

O.ool 

5  54 

61 

#  of 

re I evant  items  : 

2 

#  of 

partially  relv  « 

3 

*  of 

not  relevant  : 

34 

TOTAL  # 
NEVL 

8 


TOTAL  » 
RETRIEVED 

159 


61 


to'ca  1 

#  evaluated 

39 

#    not  eveftluated 

8 

d 

total 

#  retrieved 

■ 

39 

recal 1 

■ 
• 

0.714 

precision 

■ 
• 

d,l2S 

total 

#  of  search  terms 

■ 
• 

4 

total 

#  of  commands 

■ 
■ 

9 

total 

#  of  cycles 

■ 
■ 

2 

on-1 ine  connect  time 

8 

0.065 

off  preparation  time 

■ 
■ 

0.117 

total  search ing  ti^me       :  d •  i S2 


374 


Searoher  Not  024 


QUESt-bBASE 
NUMBER 

010154 


TOTAL  # 
RELV 


TOTAL  # 
PREL 

15 


TOTAL  « 
NREL 

130 


TOTAL  # 
EVAL 

149 


TOTAL  « 
NEVL 

310 


TOTAL  « 
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0.  009 

1.  bbo 

3 
8 
2 

0.192 
0.417 


total  searching  time 


0.609 


034013 


lb 


39 


ibb 


149 


207 


356 


#  of  relevant  items 

#  of  partially  relv 

#  of     not  relevant 


1 

13 
7 


"total  #  evalaated 

z 

21 

#     not  evaluated 

■ 

■ 

b 

total  #  retrieved 

• 
• 

21 

reoa 1 1 

■ 

• 

b.286 

preoifiion 

t 

b.667 

tota 1  #  of  search  terms : 

5 

total  #  of  oommands 

I 

5 

total  #  of  cycles 

2 

on-line  connect  time 

■ 

b.b69 

off  preparation  time 

1 

b.4l7 

total  searching  timb 

1 

b.486 
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«  of 
#  of 
«  of 

relevant  items  8 
partially  relv  s 
not  relevant  i 

12 
4 
6 

total  #  evaluated  i 
#    not  evai Dated  i 

22 
0 

total 

#  retrieved  s 

22 

rebail  s 
precision  i 

Qi31A 
0i727 

total 
total 
total 

#  of  search  terms: 

#  of  commands  i 

#  of  cycles  s 

10 
14 
3 

on-line  connect  tinie  s 
of f  preparation  time  : 

D.219 
□  .250 

total 

searching  time  : 

0i469 
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QUESTrOBASE 
NUMBER 


023032 


TOTAL  # 
RELV 

29 


tdtAt  # 
PREI. 


TOTAb  « 
NREL 


TOTAb  # 
EVAb 


9  51 

89 

#  of 

relevant  iteme  i 

1 

«  of 

partially  relv  i 

1 

«  of 

not  re 1 evant  i 

total 

#  evaluated  t 

6 

#    not  evaluated  s 

0 

total 

#  retrieved  i 

6 

recall  i 

0.052 

precision  i 

0.333 

total 

#  of  search  terms: 

total 

#  of  commands  i 

12 

total 

#  of  cycles  s 

4 

on-line  connect  time  s 

0.157 

of  f  preparation  tifhe  s 

0.033 

total  search ing  'time       t  0.160 


TOTAL  # 
NEVL 


TOTAb  » 
RETRIEVED 

S9 


037016 


78 


18 


150 


69 


219 


#  of  relevant  items        t  3 

#  of  partially  relv         t  O 

#  of     not  relevant         t  2 


total  #  evaluated 
#    not  evaluated 


5 
O 


total  #  retrieved 

■ 

m 

5 

recal 1 

i 

0.031 

precision 

m 
m 

0.600 

total  «  of  search  terms 

m 
m 

17 

total  #  of  commands 

m 
m 

39 

total  #  of  cycles 

t 

9 

on-line  connect  time 

i 

0.869 

off  preparation  time 

i 

0.333 

total  searching  time 

1 

1.202 
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QUEST-DBASE 
NUMBER 


036061 


TOTAL  « 
RELV 

79 


TOTAL  # 
PREL 

29 


TOTAL  « 
NREL 

42 


TOTAL  # 
EVAL 

150 


TOTAL  # 
NEVL 

595 


tOTAL  « 
RETRIEVE! 

745 


#  of  relevant  items 

#  of  partially  relv 

#  of     not  relevant 


6 
9 
19 


total  #  evaluated 
#    not  evaluated 


34 
240 


total  #  retrieved 

reoa 1 1 
precis ion 

total  #  of  search  terms 
total  #  of  commands 
total  #  of  Cycles 

on-line  connect  time 
off  preparation  time 


274 

6.139 
0.441 

10 
20 
5 

6.312 
6.417 


total  searching  time 


0.729 


039015 


26 


38 


lo: 


166 
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#  Of  relevant  items 

#  of  partially  relv 

#  of     not  relevant 


total  #  evaluated 
#    not  evj'luated 


4 
3 
7 


14 
0 


total  #  retrieved 

reoa 1 1 
precision 

total  #  of  search  terms 
total  #  of  commands 
total  #  of  cycles 

on— line  connect  time 
off  preparation  time 


14 

6.109 
6.500 

12 

13 
3 

0.291 
0.563 


total  searching  time       i  0.874 
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PREi.  NREi- 

EVAL 

40  32 

149 

«  of 

relevant  items 

i 

2 

«  of 

partial ly  relv 

1 

4 

«  of 

not  relevant 

1 

32 

total 

#  evaluated 

s 

36 

#    not  evaluated 

i 

4 

total 

#  retrieved 

• 

42 

reoai 1 

s 

precision 

s 

total 

#  of  search  terms s 

15 

total 

#  of  commands 

s 

13 

total 

#  of  cycles 

s 

5 

on-i  ine  connect  time 

i 

Si  293 

off  preparation  time 

i 

□  i5D0 

total 

searching  time 

I 
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TOTAb  « 
REtV 

26 


TOTAt  « 
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43 


TOTAL  « 
NREL 

81 


TOTAL  # 
EVAL 
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TOTAL  # 
NEVL 
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TOTAL  « 
RETRiEVEC 
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#  of  relevant  i'teitis 
II  of  partially  re.\y 

#  of     not  relevant 


2 
2 
6 


total  #  evaluated 
If    not  evaluated 


10 
8 


total  #  retrieved 

recal 1 
preoisibh 

total  II  of  search  terms 
total  #  of  bbrnmahds 
total  H  of  cycles 

on— line  connect  tiw^ 
off  preparation  time 


18 

0.058 
0.400 

4 
10 
2 

0.133 
0.083 


total  searching  time       s  0.216 


135  0 

150 

#  of  relevant  items 

a 
m 

i 

#  of  partially  rely 

t 

5 

#    of     not  relevant 

•i 
m 

0 

total  #  evaluated 

t 

6 

M    not  evaluated 

m 
m 

6 

total  #  retrieved 

■ 

m 

14 

recal 1 

i 

0.040 

precision 

s 

1  .000 

tbt^:!  9  of  search  terms: 

8 

total  #  of  bbmmahds 

• 
• 

9 

total  #  of  cycles 

ii 
• 

2 

bri-^lihe  bbhhebt  time 

s 

0.174 

off  preparatibh  time 

t 

0.083 

tota 1  sea rch  i ng  t ime 

t 

0.257 
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113 


44 


w        TOTAL  w       TOTAL  W 

TQTSL 

PREL  NREL 

EVAL 

19  18 

150 

#  of  relevant  items 

1 

58 

#  of  partially  re Iv 

s 

2 

#    of    not  relevant 

1 

n 
U 

tota 1  #  eva 1 uated 

i 

60 

#    not  evaluated 

i 

119 

total  #  retrieved 

s 

179 

reoai 1 

i 

0.455 

precision 

1 

1  .000 

total  #  of  search  terms: 

6 

total  #  of  commands 

1  7 

total  #  of  cycles 

I 

3 

dn~~line  connect  time 

m 

U  •  1 

off  preparation  time 

1 

total  searching  time 

m 
m 

0.246 

57  49 

150 

#  of  relevant  items 

m 
m 

15 

#  of  partially  relv 

« 
m 

26 

#    of     not  relevant 

m 
m 

o 

total  #  evaluated 

z 

47 

#    not  evaluated 

I 

38 

total  #  retrieved 

« 
m 

85 

recal 1 

■ 
m 

b  4b^i 

W  ■  "TWO 

precision 

■ 
m 

0.872 

total  #  of  search  terms 

'  3 

6 

total  #  of  commands 

3 

13 

total  #  of  cycles 

■ 

m 

4 

on'-line  connect  time 

■ 
m 

0.217 

off  preparation  time 

1 

0.083 

total  searching  time 

1 

0.300 

TOTAL  # 
NEVL 
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TOTAL  « 
RETRIEVE 

448 


123 
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0351 5 A 


TOTAt  « 
REtV 

10 


TOTAh  « 


TOTAL  #       TOTAL  # 


31 


PREL  NREL 

EVAL 

39  100 

1*9 

#  of  relevant  items 

0 

#  of  partial ly  rely 

1 

#    of     tibt  relevant 

0 

total  #  evaluated 

- 

1 

#    not  evaluated 

0 

total  #  retrieved 

1 

rebal 1 

d.020 

preoisibn 

l.OOO 

total  #  of  searbh  terms: 

h 

total       bf  bbmmahds 

12 

tbtal  #  of  oyoles 

i 

5 

bn~l in^  bbnnebt  time 

0.211 

bff  preparation  time 

0.083 

tot a 1  sea roh i ng  time 

0.294 

« 

20  14 

65 

#  of  relevant  items 

m 
m 

9 

#  of  partially  rely 

m 
m 

10 

#    bf     hot  relevant 

m 
m 

3 

tbtal  #  evaluated 

: 



22 

#     hbt  evaluated 

■ 
■ 

0 

total  #  retrieved 

22 

reba 1 1 

■ 
■ 

0.373 

preoisibn 

0.864 

tbtal  #  bf  searbh  terms 

• 
■ 

6 

tbtal  ff  of  bbmmahds 

s 

15 

tbtal  #  of  bybles 

m 

5 

bh-lihe  bbhhebt  time 

i 

0.184 

bff  preparatibh  time 

i 

0.083 

total  searching  time 

1 

0.267 

TOTAL  # 
NEVL 
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TOTAL  i 
RETRIEVE 
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TOTSL  « 
REbV 

37 


TOTAL  « 
PREL 

36 


TOTSL  « 
NREL 

156 


TOTAL  « 
EVAL 

229 


TOTAL  # 
NEVL 


TOTAL  » 
RETRIEVED 

229 


II  of  relevant  items 

#  of  partially  rely 

#  of     not  relevant 


1 
1 
17 


total  #  evaluated 
#     hot  evaluated 


19 
O 


total  #  retrieved 

recal 1 
preciBibh 

tbt^l  9  of  search  terms 
total  #  of  cbmmahds 
total  #  of  bybles 

bn-lihe  cbhhect  time 
bff  preparatibh  time 


19 

0.027 
0.105 

16 
23 
6 

0.231 
0.133 


total  searching  time       :  0.36A 


OOS064 


36 


A7 


68 


151 


27: 


423 


M  bf  releyaht  items        i  O 

#  of  partially  rely         i  2 

#  bf     not  relevant         i  O 


total  #  evaluated 
#    not  evaluated 


2 
3 


total  #  retrieved 


recal 1 
precision 


£  O . 02A 

s  i.obo 


tbtal  9  bf  search  terms:  11 
tbtal  #  bf  commands  :  16 
tctal  #  cf  cycles  ;  4 


bn-lihe  bonnect  time 
bff  preparatibh  time 


I  0.167 
i  6.100 


total  searching  time      s  0.267 
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TOTAt  # 

REbV 

18 


Searcher  Noi  033 
tOTAt  #      TOTAL  # 


TOTAL  #      TOTAL  #      TOTAL  • 


35 


PREL  NREL 

EVAL 

NEVL 

RETRIEVEl 

48  84 

150 

495 

645 

#  of  relevant  items  : 

#  of  partially  rely  t 

#  of    not  relevant  t 

1 
5 
0 

total  #  evaluated  i 
#    not  evaluated  i 

_s 

total  #  retrieved  i 

32 

rebal 1  t 
precision  i 

0.090 
1.000 

total  #  of  searcsh  terms i 
total  #  of  bbmmahds  i 
total  #  bf  cycles  t 

11 
9 
3 

bn-lihe  bbhnect  time  i 
bff  preparatibh  time  i 

0.143 

0.066 

total  searching  time  : 

0.209 

71  51 

157 

2 

159 

#  bf  relevant  items  : 

#  bf  partially  rely  i 

#  bf     hbt  relevant  t 

24 
50 
14 

tbtal  #  evaluated  s 
#    hbt  evaluated  t 

88 
1 

total  #  retrieved  s 

89 

recal 1 
preoisibh 

total  #  of  search  terms 
total  #  of  bbmmahds 
tbtal  #  bf  cycles 

bh^lihe  cbhhebt  time 
bff  preparatibh  time 


0.696 
0.841 

11 
10 
4 

0.138 
0.066 


total  searching  time       t  0.204 
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RELV 


TOTAL  # 
PREL 


TOTAL  # 
NREL 


TOTAL  « 
EVAL 


TOTAL  # 
NEVL 


TOTAL  » 
RETRIEVED 


021037 


19 


77 


102 


102 


025001 


30 


#  of  relevan'b  i^ems 

11 

#  of  partially  rely 

3 

#    of     not  relevant 

31 

total  #  evaluated 

45 

#    not  evaluated 

d 

total  #  retrieved 

45 

reca 1 1 

0.560 

precision 

0.311 

total  #  of  search  terms s 

14 

total  #  of  dommands 

2b 

6 

on-line  connect  time 

0.220 

of  f  preparation  time 

\ 

0.066 

total  searching  time 

t 

0.286 

26  94 

150 

#  of  relevant  items 

1 

a 

#  of  partially  relv 

r 

0 

#    of     not  relevant 

s 

d 

total  #  evaluated 

s 

0 

#     not  evaluated 

X 

1 

total  #  retrieved 

■ 

1 

reoal 1 

■ 

m 

diddo 

precision 

m 

OiOOO 

total  #  of  search  terms 

m 

16 

total  #  of  commands 

m 

15 

total  #  of  cycles 

I 

5 

on— line  oonnect  time 

t  0i224 

off  preparation  time 

I 
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total  searching  time 

t 
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#  of  relevant  i^ems  i 

#  of  partially  relv  i 

#  of     not  relevant  i 

16 
2 
6 

total  #  evaluated  : 
#    not  evaluated  x 

24 
0 

total  #  retrieved  : 

24 

reoa 11  : 
precision  s 

Qi474 
0.750 

total  #  of  search  terms: 
total  #  of  commands  : 
total  #  of  cycles  i 

8 

11 
3 

on-line  ddnnecst  time  : 
off  preparatidh  time  r 

0.163 
0.083 

total  searching  time  : 

0.266 

23  39 

67 

#  df  relevant  items  s 

#  of  partially  relv  x 

#  df     ndt  relevant  : 

1 
1 
12 

tdtal  #  evaluated  t 
#     ndt  evaluated  s 

14 
0 

total  #  retrieved  : 

14 

reca 11  : 
precision  : 

0.071 
0.143 

tdtal  #  df  search  terms 
tdtal  #  df  ddmmands 
tdtal  #  df  dydles 

dn-line  ddnnedt  time 
dff  preparatidh  time 


_8 
31 
7 

0.212 
0.083 


67 


total  seardhihg  time       i  0.295 
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038061 


79 


of  relevant  items 

1 

16 

of  partially  rely 

i 

1 

#    of     hot  relevant 

i 

6 

total  #  evaluated 

1 

23 

#     not  evaluated 

s 

2 

total  #  retrieved 

s 

25 

reaa 1 1 

s 

0.177 

preoisioh 

• 
■ 

0.739 

total  #  of  search  terms s 

7 

total  #  of  bbmmands 

8 

25 

total  #  of  byoles 

S 

4 

on-line  connect  time 

S 

0.220 

of  f  preparation  time 

1 

0.167 

total  searching  time 

» 

m 

0.387 

29  42 
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#  of  relevant  items 

S 

#  of  partially  rely 

1 

#     of     hot  re 1 evaht 

4 

total  #  evaluated 

13 

#     hot  evaluated 

32 

total  #  retrieved 

45 

reba 1 1 

0.083 

precision 

0.692 

total  #  of  searbh  terms 
total  #  of  bbmmahds 
total  #  of  cycles 

on- line  bbhtiebt  time 
bff  preparatibn  time 


4 
32 
6 

0.303 
0.083 
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total  searching  time       i  0.386 
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#  of  reievssnt  items 

#  of  partially  rely 

#  of     not  relevant 


total  #  eyaiaated 
#    not  evaluated 


2 
5 
5 


12 
O 


total  #  retrieved 

recal 1 
precision 

total  #  of  search  terms 
total  #  of  commands 
total  #  of  cycles 

on-line  connect  time 
off  preparation  time 


12 

□  .109 
0.583 

7 
17 

0il83 
Si  083 


total  searching  time 


0.266 


040016 


77 


4d 


32 


149 


331 
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#  of  relevant  items         s  6 


#  of  partially  relv  : 

#  of    not  relevant  : 

2 
0 

total  #  evaluated  t 
#    not  eveiluated  s 

8 

7 

total  #  retrieved  : 

15 

reca 1 1  i 
precision  t 

0i068 
1  iOOd 

total  1  #  of  search  terms: 
total  #  of  commands  t 
total  #  of  oyoies  i 

IS 
32 
8 

on-line  connect  time  i 
off  preparation  time  i 

Si  367 
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total  searohing  time       i  0.617 
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NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVE 
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#  of  relevant^  ±t^m&         t  3 

#  of  partially  re Iv         t  4 

#  of     hot  relevant        t  28 


total  #  evaluated 
#    hot  evaluated 


35 
0 


total  #  retrieved 


35 


reoai i  

preoisibh 


I  0.164 
c  0.200 


total  #  of  search  terms s  6 
total  #  of  oocnmands  s  8 
total  #  of  oyoies  i  3 

oh-lih©  oonnect  time  s  0^117 
off  preparation  time       s  0.167 


tota 1  search ihg  time 
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037016  78  18  54  150  69  219 

#  of  relevant  items  :  29 

#  of  partially  rely  s  1 

#  of     not  relevant  s  6 


total  #  evaluated  s  36 

#    hot  evaluated  :  19 

total  #  retrieved  s  55 

recall  i  0^313 

precision  :  0.833 

total  #  of  eeardh  terms:  9 

total  #  of  dpmmands  :  17 

total  #  of  cydles  s  5 

on-line  dohhedt  time  :  0^252 

off  preparatibh  time  t  0^083 


total  searching  time       t  0.335 
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TOTAL  « 
REi-V 
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TOTAt  « 
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#  of  partially  rely 

#  of     hot  Ire  levant 


total  #  evaluated 
#    hot  evaluated 


total  #  re'trieved 
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total  #  of  searcsh  terms 
total  #  of  oommahds 
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6h-lihe  bbnhect  time 
off  preparatibh  time 


total  searching  time 


6 
2 
2 


ID 
15 


25 

6.07774 
0.800 

13 
19 

4 

6.278 
6.083 


0.361 
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26 


38 
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#  of  relevant  items        t  O 

#  of  partially  felv        <  0 

#  of     hot  relevant        i  6 


total  #  evaluated 
#    hot  evaluated 
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6 


total  #  retrieved 

reca 1 1 
precisioh 


total  #  of  search  terms: 
total  #  of  bbmmands  : 
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oh-lihe  bbhhebt  time  i 
off  preparatibh  time  i 
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0.000 
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16 
5 
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15 
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■ 

A7 

total  #  retrieved 

62 

recall 

: 

0.128 

precision 

1 

1  .000 

total'  #  of  searoh  terms i 

6 

total  #  of  commands 

: 

14 

total  #  of  cycles 

1 

6 

on-line  connect  time 

1 

Q.2B7 

off  preparation  time 

1 

0.033 

tota 1  search ing  time 

1 

0.340 

ERIC 


406 


Sttsrohttr  Ns;  036 


QUESt-DBASE 
NUMBER 

024191 


TOTAL  # 
RELV 


TOTAL  « 
PREL 


TOTAL  # 
NREL 

3b 


TOTAL  # 
EVAL 

35 


TOTAL  « 
NEVL 


TOTAL  9 
RETRIEVEI 

35 


#  of  relevant  items  t  1 
«  of  partially  rely         i  2 

#  of     not  relevant         t  8 


total  #  evaluated 
#    not  evaluated 


11 
O 


total  :#  retrieved 


11 


recal 1  

precision 


I  0.273 


total  #  of  search  terms:  17 
total  #  of  commands  i  l4 
total  #  of  cycles  t  3 

on-line  connect  time  i  bi362 
off  preparation  time  ^  0ill7 

total  searching  time       :  d.A79 


026038 


37 


39 


B 


8A 


8A 


#  of  relevant  items 

1 

#  of  partial iy  reiv 

• 
• 

1 

#    of     not  relevant 

8 

0 

tcstai  #  evaluated 

8 

2 

#    not  evaluated 

8 

0 

total  #  retrieved 

8 

2 

reoal 1 

• 

6^026 

preci&ion 

• 

1 .000 

total  #  of  search  terms: 

9 

total  #  of  commands 

• 

8 

total  #  of  oyoies 

■ 
• 

2 

on-linf?  connect  time 

1 

0.195 

off  preparation  time 

: 

OiSOO 

total  searohing  time       8  d.495 


407 


QUESt-bBASE 
NUMBER 
028038 


030071 


Searcher  No.  03£ 
t©tAL  «       TOTAL  «       TOTAL  # 
RELV 
5 


TOTAL  * 


57 


SVAt 

23  39 

67 

#  of  relevant  items 

: 

0 

i  of  partially  rslv 

d 

#    of     not  relevant 

s 

G 

total  #  evaluated 

s 

b 

#    not  evaluated 

s 

d 

total  #  retrieved 

8 

b 

rebal 1 

s 

b.dbb 

precision 

s 

b.dob 

total  #  of  search  terms s 

ii 

total  #  of  commands 

8 

ii 

total  #  of  oyoles 

s 

2 

on-line  connect  time 

1 

d.265 

off  preparation  time 

s 

b.i67 

tota 1  search ing  time 

: 

d.432 

25  13 

9S 

#  of  relevant  i1^,ems 

: 

i 

m  of  partially  relv 

: 

d 

#    of     not  relevant 

m 

m 

d 

total  #  evaluated 

1 

i 

#    not  evaluated 

: 

d 

total  #  retrieved 

8 

1 

reca 11 

• 

m 

precision 

m 
• 

i^ddo 

total  «  Df  search  terms: 

15 

total  #  of  commands 

8 

11 

total  #  Df  cycles 

8 

3 

on-line  connect  time 

8 

S.2S7 

off  preparation  time 

8 

0^483 

total  searching  time 

8 

d.74D 

408 

tOTAb  #      TdTAfc  » 
NEVb  REtRiEVEC 
□  67 


95 


f 


Searcher  No;  036 

QUEST-DBASE       tOtAL  «       TOTAL  #       TOTAL  «  TOTAL  #  TOTAL  «       TOTAL  • 

NUMBER               RELV             PREt            NREL  2VAL  NEVL  RETRIEVEC 

03106i                  14                is                85  114  O  114 


#  oi  relevant  items  s  3 
^  of  fDartialiy  relv         i  0 

#  of     not  relevant         s  8 


total  #  evaluated 
#    not  evaluated 

8 
8 

11 

b 

total  #  retrieved 

8 

ii 

reoa^  i 
precision 

• 
• 

■ 
• 

o.ibs 

0.273 

total  #  of  search  terms s 
total  #  of  commands  s 
total  #  of  cycles  3 

4 
4 

2 

on-line  connect  ti^me 
off  preparation  time 

• 
• 

8 

0.104 
0.233 

tota i  search i ng  time 

• 
• 

0.337 

49  39 

150 

17  207 


#  cf  relevant  items 

• 
• 

28 

#  of  partially  relv 

■ 
• 

9 

#    of     not  relevant 

■ 
■ 

Id 

total  #  evaluated 

• 
* 

47 

#    not  evaluated 

■ 

0 

total  #  retrieved 

• 

m 

47 

recal 1 

m 

m 

0i333 

precision 

m 
m 

0.7787 

total  #  of  search  terms 

•  S 

4 

total  #  of  commands 

■ 
a 

7 

total  #  of  cycles 

1 

2 

on-~l  ine  connect  time 

i 

0ii57 

off  preparation  time 

t 

0.083 

total  searching  time 

1 

0i240 

409 


Searcher  No.  037 


QUEST-DBASE  TOTAb 
NUMBER  REtV 

004154  60 


#      TOTAb  #      TOTAb  # 

TOTAL 

PREb  NREb 

EVAL 

58  33 

151 

#  of  relevant  items 

i 

12 

1 

#    of     not  felevantr 

i 

io 

total  #  evaluated 

i 

Off 

If    hot  evaluated 

8 

total  #  retrieved 

8 

36 

reoall 

8 

0.203 

preoisioh 

8 

0.706 

total  $  of  %fBSiroh  terms s 

•5 

total  #  of  bbmrnahds 

8 

12 

total  #  of  oybles 

8 

5 

on-line  dbnhebt  time 

8 

6.168 

off  preparation  time 

8 

0.066 

total  searching  time 

8 

0.234 

TOTAb  # 
NEVb 

8 


TOTAb  <» 
RETRIEVE 

1S9 


008154  2  5  54  61  O  61 


#  of  relevant  items 

8 

2 

#  of  partially  relv 

8 

0 

#    of     libt  relevant 

• 
■ 

11 

tbtal  #  evaluated 

8 

13 

#    not  evaluated 

t  0 

total  #  retrieved 

8 

13 

reca 1 1 

• 

a 

0.266 

precision 

• 

0.154 

tbtal  41  of  search  terms s 

4 

tbtal  #  of  bbmrnahds 

8 

14 

tbtal  41  of  bybles 

• 

4 

bn-lihe  bonnebt  time 

t 

0.111 

off  preparation  time 

t 

0.133 

total  searching  time 

t 

0,244 

8«aroh«r  No.  037 


QUEST-DBASE 
NUMBER 


tOTAt  « 
RELV 


TOTAL  # 
PREL 

15 


TOTAL  « 
NREL 

130 


TOTAL  # 
EVAL 

149 


TOTAL  # 
NEVL 

310 


TOTAL  « 
RETRIEVED 

459 


#  of  relevant  i tents 

#  of  partially  relv 

#  of     not  relevant 


2 

i 

2 


total  #  evaluated 
#    riot  evaluated 


5 
13 


total  #  retrieved 

redal 1 
preoiBidn 

total  #  of  jseafoh  terms 
total  #  of  commands 
total  #  of  oyoles 

on-line  connect  time 
off  preparation  time 


18 

0.158 
0.600 

7 

10 
2 

0.136 
0.200 


total  Searching  time 


0.336 


011154 


25 


115 


149 


161 


310 


#  Of  relevant  items  e  O 

#  of  partially  relv  s  1 

#  of    not  relevant  i  1 

total  #  evaluated  i  2 

#  riot  evaluated  s  3 

total  #  retrieved  :  5 

recall  l  0.029 

precision  :  0.500 

total  It  of  search  terms:  8 

tdt^iJ  #  Of  commands  :  9 

total  It  of  cycles  :  3 

orl-line  bonnect  time  s  0^083 

off  preparation  time  s  bil67 

total  searching  time  i  0.250 


8esroh«r  Ne.  037 


QUEST-DBASE 
NUMBER 

015154 


tOTAt  # 

REtV 

28 


TOTAt  « 
PREt 

86 


TOTAt  « 
NREL 

36 


TOTAL  « 
EVAL 

150 


TOTAL  # 
NEVL 

172 


TOTAL  « 
RETRIEVEi 

322 


«  of  relevant  items 

#  of  partially  rely 

#  of    not  relevant 


tota 1  #  eya 1 uated 
#    not  evaluated 


13 
21 
9 


43 
52 


total  #  retrieved 

reoal 1  

precision 

total  #  of  search  terms 
total  #  of  commands 
total  #  of  cycles 

on-line  connect  time 
off  preparation  time 


95 

bi298 
0.791 

3 
9 
2 

0.234 
0.133 


total  searching  time 


0.367 


8«aroher  No.  038 


QUEST-DBASE 
NUMBER 

006006 


TOTAL  # 
RELV 

11 


TOTAL  # 
PREL 


TOTAL  « 
NREL 

134 


TOTAL  « 

evaL 

150 


TOTAL  « 
NEVL 

68 


TOTAL  » 
REtRIEVED 

238 


#  of  rvlevan't  i^ems 

#  of  partially  rel^v 

#  of     not  relevant 


total  #  evaluated 
#    not  evaluated 


b 
d 

2 


2 
1 


total  #  retrieved 

reoal 1  

precision 

total  #  of  eearoh  terms 
total  #  of  ooinmands 
total  #  of  cycles 

on-line  connect  time 
off  preparation  time 


0^000 
O^dOO 

13 
18 
4 

0i036 
0i250 


total  searching  time 


0i286 


017005  36  26  88  150  377  527 

#  of  relevant  items  i  O 
■  #  of  partial iy  relv  i  1 

#  of     not  relevant  :  0 

total  #  evaluated  :  1 

#  not  evaluated  :  0 

total  #  retrieved  x  1 

recall  t  0.016 

precision  :  liOOO 

total  #  of  search  terms:  9 

total  #  of  commands  i  13 

total  #  of  cycles  :  3 

on-line  connect  time  :  0.095 

off  preparation  time  t  0.217 

total  searching  time  :  0.312 


ERIC 


413 


Searcher  No.  D3S 


QUEST-DBASE 

TOTAL 

#       TOTAL  #       TOTAL  # 

TOTAL 

PREL  NREL 

CVAL. 

022108 

IS 

135  0 

150 

m  of 

relevant  items 

■ 
1 

1 

«  of 

partial ly  rel v 

i 

5 

#  of 

not  relevant 

i 

b 

total 

#  evaluated 

i 

O 

#    not  evaluated 

s 

11 

total 

#  retrieved 

i 

1  f 

recal 1 

i 

b.b^b 

precision 

i 

1  .000 

total 

#  of  search  terms s 

lb 

total 

#  of  commands 

s 

18 

total 

#  of  cycles 

s 

6 

on-li 

he  connect  time 

m 

0.148 

of  f  preparation  time 

LJ  •  1 0  ^ 

total 

sea  r ch  i  ng  t  i  me 

1 

0.315 

029008 

36 

34  60 

150 

«  of 

re 1 eyant  items 

: 

«  of 

partial ly  relv 

: 

1 

«  of 

not  relevant 

: 

1 

total 

#  evaluated 

8 

#    not  CEfvaluated 

: 

3 

total 

#  retrieved 

• 
• 

recal 1 

: 

0.057 

precision 

! 

0.800 

total 

M  of  search  tei.Tis 

>  8 

9 

total 

#  of  oommahds 

8 

17  • 

total 

#  of  cycles 

8 

6 

on-line  connect  time 

8 

0.326 

of  f  preparation  time 

8 

0.167 

total 

sea  r ch  i  ng  t i  me 

1 

0.493 

TOTAL  # 
NEVL 

365 


TOTAL  « 
RETRIEVE! 

515 


77 


227 


4i4 


Scaroher  No*  Q39 


QUEST-DBASE       TQTAb  #       TOTAL  «       TDTSU  «       TPTSU  «       TOTAL  «       TOTAL  1 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVE! 

006006  11  5  134  150  6B  23B 


#  of  relevant  iteffie 

1 

2 

41  of  partially  relv 

• 
« 

#    of     not  relevant 

i 

3^ 

total  #  evaluated 

s 

#    hot  evaluated 

8 

ZD 

total  #  retrieved 

t 

65 

redal 1 

: 

0.375 

precision 

i 

0.133 

total  #  of  Heardh  terms s 

lb 

total  41  of  cbinfhahds 

i 

21 

total  #  of  cycles 

t 

6 

on-line  ddnnedt  time 

i 

-  -   

0.250 

off  preparation  time 

t 

0.250 

total  searching  time 

% 

m 



0.500 

36                26  66 

150 

#  of  relevant  items 

I 

22 

#  of  partially  relv 

t 

9 

#    of     not  relevant 

t 

o 

tota 1  #  eva 1 uated 

t 

39 

#    not  evaluated 

t 

94 

tota 1  #  retrieved 

1 

133 

recal 1 

■ 
m 

0.500 

precision 

m 
m 

0.795 

total  #  of  search  terms 

12 

total  #  of  commands 

■ 

18 

total  #  of  cycles 

m 
m 

3 

on-iine  connect  time 

t 

0.301 

dff  preparation  time 

t 

0.250 

total  searching  time 

0.551 

017005  36  26  88  150  377  527 


415 


ERIC 


Searcher  No.  039 


QUEST-DBASE       tOTAL  #       TOTAL  «       TOTAL  #       TOTAL  #       TOTAL  #       TOTAL  « 
NUMBER  RELV  Pf?EL  NREL  EVAL  NEVL  RETRIEVE! 

022108  15  135  O  150  365  515 

#  of  relevant  items  i  O 

#  of  partially  rely  i  6 
#  of  not  relevant   i  0 

tota 1  #  ey a 1 aa ted  i  6 

#  not  evaiaated  s  64 


total  #  retrieved  i  70 

reoa 1 1   i  0  ^  040 

precision  s  i .Odd 

total  #  of  search  terms:  9 

total  #  of  commands  r  26 

total  #  of  cycles  t  6 

on-iine  connect  time  s  0.306 

off  preparation  time  i  0.250 


tota 1  searching  time       s  O « 556 


029008  36  34  80  15d  77  227 


#  of  relevant  items        i  5 

#  of  partially  relv         :  3 

#  of     not  relevant         t  2 

tota 1  #  eya 1 uated  :  10 
#  not  evaluated   :  5 

total  #  retrieved  :  15 

recall  :  6*114 

precision  :  O  i 800 

total  #  of  search  terms:  9 
total  #  of  commands  :  25 
total  #  of  cycles  :  6 


EKLC 


on-line  connect  time  :  0*268 

of  f  preparation  time  i  0*^250 

total  searching  time  i  0.518 

418 


8»aroh*r  No.  040 


QUEST-DBASE 
NUMBER 

006C306 


TOTAL  # 
RELV 

ii 


TOTAL  # 
PREL 


TOTAL  « 
isiREL 

134 


TOTAL  « 
EVAL 

150 


TOTAL  « 

KiiEVL 

88 


TOTAL  tt 
REtRIEVED 

238 


#  of  relevant  items        i  O 

#  of  partially  relv        i  O 

#  of     not  relevant        t  4 


total  #  evalaated 
#    not  evalaated 


4 

d 


total  #  retrieved 

recall  

precision 


I  d.dod 
I  diddd 


total  #  of  search  terms:  13 
total  #  of  commands  i  38 
total!  #  of  cycles  i  13 

on-line  oonneot  time  i  di67d 
off  preparation  time       i  d.ll7 

total  searching  time       i  d.787 


dl2dl3 


21 


121 


148 


9d 


238 


#  of  relevant  items 

#  of  partial  iyri?lv 

#  of     not  relevant 


1 
1 

2 


tota 1  #  e va 1 aat ed 
#    not  evaluated 


4 
3 


total  #  retrieved  :  7 

recall   :  did74 

precision  t  d^Sdd 

total  #  of  search  terms:  61 
total  #  of  commands  t  41 
total  #  of  cycles  t  6 

on-line  oonnect  time  t  d^723 
off  preparation  txme  :  d^583 

total  searching  time       t  1.3d6 


ERIC 


417 


SNrcher  No.  040 


QUESt-bBASE       TOTAL  #       tOTAL  ft       TOTAL  #       TOTAL  #       TOTAL  «       TOTAL  9 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVEI 

017005  36  26  88  150  377  527 


#  of  relevant  items 

#  of  partial iy  reiv 

#  of     not  relevant 


tot a 1  #  ey a 1 aated 
#    not  evaluated 


7 
6 
19 


32 
75 


total  #  retrieved 

reoal 1  

precision 

total  #  of  search  terms 
total  #  of  commands 
total  1f  of  cycles 

off  preparation  time 


107 

0^210 
0.406 

9 

21 
3 

0.4Q9 

d«il7 


tota 1  searching  time       :  O . 526 


027013 


35 


69 


46 


150 


58 


208 


#  of  relevant  items        s  19 

#  of  partially  reiv        i  14 

#  of     not  relevant        i  2 


total  #  evaluated 
#    hot  evaluated 


35 
12 


tota 1  #  retrieved 


47 


recal 1 
precision 


:  0.317 
:  0.943 


total  #  of  search  terms:  10 
total  #  of  commands  :  23 
total  #  of  cycles  :  6 

oh--iihe  cohheot  time  :  0.362 
off  preparation  time       :  Di066 

total  searching  time       i  0i428 


erJc 


418 


8eairohte»r  No.  040 


QUESTrCBASE 
NUMBER 

029008 


TOTAL  # 
RELV/ 

36 


TOTAL  # 
PREL 

34 


TOTAL  # 
NREL 

6b 


TOTAL  # 
EVAL 

150 


TOTAL  # 
NEVL 

77 


TOTAL  « 
RETRIEVE 

227 


#  of  reieyan-b  items 

#  of  partially  relv 

#  of     not  relevant 


total  #  evaluated 
#    not  evaluated 


3 
0 

b 


3 

b 


total  #  retrieved 

recall 
preoieion 

total  #  of  search  terms 
total  #  of  commands 
total  #  of  cycles 

on-line  ocnneot  time 
off  preparation  time 


0.  042 

1.  bbb 

22 
30 
9 

0.423 
0.333 


total  searching  time       i  0.756 


Searoher  No.  041 


QUEST-DBASE       tOtAt  #       tOTAt  #       tOtAt  #       TOTAt  «       TOtAL  # 


NUMBER 

RELV 

PREL  NREL 

EVAi- 

NEVL 

023032 

29 

9  Si 

89 

6 

• 

«  of 
«  of 
«  of 

relevant  items  i 
partially  relv  i 
not  relevant  i 

13 
0 
6 

-total  #  evaluated  i 
#    not  eval dated  s 

21 
O 

total 

#  retrieved  i 

recall  t 
precision  s 

21 

0i342 
0.619 

tota  1 
total 
total 

#  of  search  ternisi 

#  of  oonifliands  i 

#  of  cycles  i 

5 
14 
4 

on-line  connect  time  i 
off  preparation  time  r 

n  -  4  AO 
0i25e 

total 

searching  time  t 

0i439 

03701 h 

78 

18  54 

150 

69 

«  of 

#  of 

#  of 

relevant  items  t 
partially  relv  i 
not  relevant  r 

31 
6 
10 

tota 1  #  eya 1 uated 
#    not  evaluated 


47 
26 


total  #  retrieved 

recal 1 
precision 


total  #  of  search  terms: 
total  #  of  commands  : 
total  #  of  cycles  r 


73 

0.385 
0.787 

7 
13 
4 


on-line  connect  time  t  0.198 
of f  preparation  time       t  0.167 

————————¥——— T———_ 

total  searching  time       s  0.365 


TOTAL  « 
RETRIEVE 

89 


219 


4bo 


Searcher  No.  041 


QUEST-DBASE      TOTAL  #       TOTAL  tt      TOTAL  «      TOTAL  «       TOTAL  tt       TOTAL  # 
NUMBER  RELV  PREL  NREL  EVAL  NEVL  RETRIEVEl 

038061  79  29  42  150  59S  745 


«  of 

relevant  items  i 

29 

«  of 

partially  relv  i 

8 

#  of 

not  relevant  i 

mm 

3 

total 

#  evaluated  t 

40 

#    not  evatoated  8 

64 

total 

#  retrieved  x 

104 

recall   i 

0.343 

precision         '  i 

£3.925 

total 

#  of  search  terms: 

17 

total 

#  of  commands  x 

45 

total 

#  of  cycles  i 

14 

on-iine  connect  time  i 

0.817 

off  preparation  time  i 

0.417 

total 

sea  r ch  i  ng  t i  me 

1  1.2- 

□39015  26  38  102  166  O  166 


#  of 

relevant  items 

X 

1 

#  of 

partial ly  relv 

• 

0 

#  of 

not  relevant 

X 

1 

total 

#  evaluated 

X 

2 

#     not  evaluated 

X 

0 

total 

#  retrieved 

m 

2 

reca 1 I 

m 

0i0i5 

precision 

w 

0.500 

total 

#  of  search  terms 

S 

7 

total 

#  of  oomfnands 

■ 

19 

total 

#  of  cycles 

8 

4 

on-line  connect  time 

■ 

0  -.  285 

off  preparation  time 

t 

Oi4l7 

total  searching  time       i  6.702 


Searcher  Nbi  641 


QUESt-DBASE 
NUMBER 


040016 


TOTAL  # 
RELV 

77 


TOTSL  » 
PREL 

40 


TOTSL  # 
NREL 

32 


TOTAL  « 
EVAL 

149 


TOTAL  # 
NEVL 

331 


TOTAL  m 
RETRIEVED 

480 


it  of  relevant  It^emB 
it  of  partially  relv 
it    of    not  relevant 


16 
9 
0 


tota 1  #  eva 1 uated 
#    not  evaluated 


25 
18 


total  #  retrieved 

recai 1 
precision 

total  it  of  search  terms 
total  it  of  ooflimandti 
total  it  of  oycies 

on-line  connect  time 
off  preparation  time 


43 
0.214 

i.oob 

3 
19 

7 

0.321 
0.333 


total  searching  time      s  0.654 


422 


^fSNDXX  D«  ACCESSION  NUMBER  (I.E.,  ITEMS  RETRIEVED)  OVERLAP  BETWEEN 
PAIRS  0?  SEARCHERS 


Appendix  D  contains  one  page  of  data  for  each  of  the  40 
questions  oh  the  number  and  proportion  of  accession  numbers  retrieved 
In  common  (i.e.^  overlap  in  Items  retrieved)  by  all_pbs8lble  palrwise 
comblhatlpns  of  outside  searches  of  the  question.    Each  data  page  in 
Appendix  D  is  organized  into  three  sections  which  include  the 
following: 

1.  Sesch  Evaluation  ittcluding  a  definition  for  each  of  the 
headers  used  on  the  data  sheet »  A  through  J2. 

2.  Tota£^or  Question  including  the  actual  values  for  each 
search  in  each  items  retrieved  category: 

A.  Number  of  relevant  items  retrieved  by  search 

B.  Nvmber  of  partially  relevant  items  retrieved  by  search 

C.  Number  of  hot  relevant  items  retrieved  by  search  • 
p.  Number  of  evaluated  items  retrieved  by  search  _ 

E.  Ntxmber  of  hot  evaluated  items  retrieved  by  search   

F#  Total  number  of  items  retrieved  by  search   

3.  Search  Overlap  lists  the  overlap  in  accession  numbers  between 

P?^? _ outside  searches »  the  number  of  accession  

numbers  found  in  common  (i»e-i_OY?riap)  in  each  category  A. 
through  F.  is  listed.    The  data  are  used  to  calculate  the 
prbpbrtibns  br  bverlap  measures  listed  In  cblumns  Gl  tb  32 
for  each  pair  of  searches.  Definitions  of  the  proportions 
calculated  are  as  follow: 

Gl  «  Total  htmber^  of  items  retrieved  by  both  searcher  1  and  2 
Total  number  retrieved  by  searcher  1 

G2  «  Total  number  of  items-retrieved  by  l^oth  searcher  l-and-2 
Total  number  retrieved  by  searcher  2 

HI  «  Total  number  of  evaluated  items  retrieved  by  both  Si  and  82 
Total  number  bf  evaluated  items  retrieved  by  Si 

H2  »  Total  number  bf  evaluated  items  retrieved  by  both  SI  and  S2 
Total  number  of  evaluated  items  retrieved  by  S2 
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Total  nufflbnr  of  relevant  plus  partially  relevant  items 

11  -  _  retrieved  by  both  Si  and  S2  

Total  number  of  relevant  plus  partially  relevant  items 
retrieved  by  SI 

Total  number  of  relevant  plui  partially  relevant  items 

12  ■  retrieved  by  both  St  and  S2 

Total  number  of  relevant  plus  partially  relevant  items 
retrieved  by  S2 

Jl  ■  Total  number  of  not  evaluated  itema^etrleved  by  both  St  and  S2 
Total  number  of  not  evaluated  items  retrieved  by  Si 

J2  ■  Total  number  of  not  evaluated  items  retrieved  by  both  SI  and  S2 
Total  number  of  not  evaluated  items  retrieved  by  S2 


EKLC 


4P.4 


QUESTION  #  :  OOl 
DATABASE  #   :  Oil 


*     Accession  Number  Overlap  between  pairs  of   Searchers  * 


Search  EvalLiatibh 


A:  #  Relevant 

B:  #  Partially  Relevant 

C:  #  Not  Relevant 

D:  Total  #  Evaluated 

E:  #  Not  Evaluated 

F:  Total  :#  Retrieved 


***  _ Proportions  *** 

Gl :   F  in  overlap  between  S1&S2  /  F 

G2:   F  in  overlap  between  Si&S2  /  F 

His   p  in  overlap  between  S1&S2  /  D 

H2 s   D  in  overlap  between  SI &S2  /  D 

II:   A+B  in  bverlafD  between  S1&S2  / 

12 s   A+B  in  overlap  between  SI §S2  / 

Jl :   E  in  overlap  between  S15S2  /  E  for  SI 

J2:   E  in  overlap  between  S1&S2  /  E  for  S2 


for  SI 
for  S2 
for  Si 
for  S2 
A+B  for  Si 
A+B  for  S2 


Total   for  Question  : 


Searcher 

A 

B 

C 

D 

E 

F 

□02 

3 

2 

1 

6 

9 

15 

□OS 

7 

2 

□ 

9 

11 

2^ 

□  13 

S 

3 

0 

8 

28 

36 

□  16 

i 

9 

2B 

35 

□ 

35 

□21 

ii 

3G 

41 

82 

443 

525 

Searcher  Overlap 


Searcher 


SI 

S2 

A 

B 

C 

D 

E 

F 

61 

G2 

HI 

H2 

11 

12 

□02 

□□B 

1 

□ 

□ 

1 

1 

2 

□ 

- 133 

□  . 

!□□ 

□  . 

166 

g 

.111 

g. 

2Qg 

d. 

1  i  i 

002 

□13 

1 

□ 

□ 

1 

0 

□ 

.□66 

g. 

g27 

g. 

166 

g 

.125 

g. 

2GG 

d. 

125 

□□2 

□16 

0 

0 

□ 

□ 

□ 

□ 

□ 

.□□□ 

g. 

ggg 

g. 

ggg 

0 

.ggo 

□  . 

GGG 

0. 

000 

□□2 

□21 

□ 

0 

□ 

□ 

□ 

□ 

□ 

-□□□ 

□  . 

□□□ 

□  . 

g^^ 

g 

.ogg 

g. 

GGG 

0. 

000 

□□5 

□13 

i 

0 

□ 

1 

5 

6 

□ 

.3^0 

□  . 

166 

□  . 

111 

0 

.125 

g. 

111 

g. 

125 

□05 

□16 

d 

0 

0 

0 

□ 

□ 

.□□□ 

□  . 

□CDO 

□  . 

□□□ 

0 

.□□□ 

□  . 

GOG 

g. 

ggg 

□□5 

□21 

□ 

□ 

□ 

□ 

d 

0 

0 

.000 

□□□ 

□  . 

□□□ 

CJ 

.□□□ 

□  . 

GOG 

0. 

000 

□13 

□16 

□ 

□ 

0 

d 

d 

C 

.000 

0. 

□□0 

□  . 

□□0 

□ 

.□□G 

□  . 

000 

0. 

000 

□13 

□21 

g 

□ 

□ 

□ 

d 

□ 

.  □□□ 

G. 

dQG 

e. 

000 

0 

.□□□ 

□  . 

000 

0. 

000 

□16 

□21 

□ 

3 

1 

4 

□ 

4 

G 

.1  i4 

□□7 

□  . 

114 

0 

.048 

0. 

300 

0. 

073 

Jl 

0.111  o.c 
d.odo  o.e 
d.ddd  d.e 
g.ogg  o.c 
g.454  g.i 
D.ggg  g.c 
g.ggg  g.c 
g.ggg  g.c 
o.CDOO  g.c 
0.000  o.c 


i25 
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QUESTION  #  :  GG2 
DATABASE  #    :  218 

******************** 

*  Accession  Number  Overlap  between  pairs  of  Searchers  * 
********************************************************* 

Search  Evaiuatidh  : 


*** 

Proportions  *** 

A: 

#  Relevant 

Gl  : 

F  in  overlap  between  S15S2  / 

F 

for 

SI 

B: 

#  Partial ly  Relevant 

G2: 

F  in  overlap  between  S1&S2  / 

F 

for 

S2 

C: 

#  Not  Relevant 

HI  : 

b  in  overlap  between  S1SS2  / 

D 

for 

SI 

D: 

Total  #  Evaluated 

H2: 

b  in  overlap  between  S1SS2  / 

D 

for 

S2 

E: 

#  Not  Evaluated 

il  : 

A+B  in  overlap  between  S1&S2 

/ 

A+B 

for 

SI 

F: 

Total   #  Retrieved 

12: 

A+B  in  overlap  between  S1&S2 

/ 

A+B 

for 

S2 

Jl  : 

E  in  overlap  between  SiaS2  / 

E 

for 

SI 

J2: 

E  in  over! ap  between  Si &S2  / 

E 

for 

S2 

Total   for  Question  : 


Searcher       A  B 


0G3 

B 

d04 

□ 

□ 

006 

14 

4 

□26 

16 

8 

□33 

1 

1 

C 

D 

E 

F 

12 

22 

□ 

22 

2 

2 

□ 

2 

7 

25 

□ 

25 

61 

85 

0 

85 

17 

19 

0 

19 

Searcher  Overlap  : 


Searcher 

SI 

S2 

A 

B 

C 

D 

E 

F 

Gl 

G2 

HI 

H2 

11 

12 

Jl 

003 

004 

0 

0 

G 

□ 

0 

0 

0 

.000 

0 

-000 

0- 

000 

0 

-000 

0 

.CDOO 

0 

.000 

0 

.000 

0 

.0 

003 

006 

3 

0 

2 

5 

0 

5 

□ 

.227 

0 

.200 

0- 

?27 

0 

-200. 

0 

.300 

0 

.166 

0 

.000 

0 

.0 

003 

026 

3 

i 

3 

7 

b 

7 

0 

.318 

0 

.082 

0. 

318 

0 

.082 

0 

.400 

0 

.166 

0 

.000 

0 

.0 

003 

033 

0 

d 

Q 

0 

d 

0 

0 

.000 

0 

.000 

0. 

oao 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0 

.01 

004 

006 

0 

0 

0 

d 

d 

0 

0 

.000 

0 

-000 

Oi 

000 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0 

.0 

004 

026 

0 

d 

2 

2 

d 

2 

1 

.000 

0 

-023 

1 . 

000 

0 

.023 

0 

.000 

0 

.000 

0 

.000 

0 

.01 

004 

□33 

P 

0 

0 

0 

d 

d 

d 

.dOd 

d 

-ddd 

d. 

ddd 

d 

.000 

d 

.odd 

0 

.odd 

0 

.odd 

b 

.bi 

006 

□26 

10 

3 

5 

IS 

d 

18 

d 

.72d 

d 

.211 

d. 

72d 

d 

.211 

0 

.722 

d 

.541 

0 

.ddd 

0 

.01 

006 

□33 

0 

0 

0 

0 

0 

d 

d 

.dOd 

p 

-ddd 

d. 

ddd 

d 

.dOd 

0 

.000 

0 

.000 

0 

.000 

0 

.01 

026 

□33 

0 

0 

0 

0 

0 

0 

0 

.000 

0 

-000 

0. 

000 

0 

.000 

0 

.000 

0 

.oou 

0 

.000 

0 

.0( 
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QUESTION  #  :  003 
DATABASE  #   :  064 


*    J^Pps^^ip^  NMo^ber  Overly  pairs  of   Searchers  * 

Search  Evaluation  : 


■K- 

Proportions  *** 

A: 

»  Relevant 

Gl  : 

F  in  overlap  between  Si&S2  / 

F 

for 

SI 

B: 

»  Partially  Relevant 

G2: 

F  in  overlap  between  S1&S2  / 

F 

for 

S2 

C: 

S  Not  Relevant 

HI  : 

D  in  overlap  between  S1&S2  / 

D 

for 

SI 

D: 

Total  g  Evaluated 

H2: 

D  in  overlap  between  S1&S2  / 

D 

for 

S2 

E: 

#  Not  Evaluated 

11: 

A+B  in  overlap  between  Si&S2 

/ 

A+B 

for 

F: 

Total   #  Retrieved 

12: 

A+B  in  over  1  afD  between  Si  &S2 

/ 

A+B 

for 

Jl  : 

E  in  overlap  between  S1&S2  / 

E 

for 

Si 

J2: 

E  in  overlap  between  S1&S2  / 

E 

for 

S2 

Total  for  Question  : 


Searcher       A       B       6       D       E  F 


002 

9 

IS 

26 

S3 

io 

63 

014 

1 

2 

1 

4 

i2 

i6 

016 

0 

0 

0 

0 

117 

117 

021 

13 

2? 

34 

76 

85 

161 

033 

0 

2 

0 

2 

3 

5 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

e 

D 

E 

F 

Gl 

G2 

HI 

H2 

11 

12 

Jl 

u 

002 

014 

D 

1 

d 

1 

d 

i 

d 

.dis 

b 

.062 

0 

.018 

0 

.210 

0 

.037 

0 

.  333 

0 

-  oob 

0. 

be 

002 

016 

0 

0 

d 

d 

d 

0 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0. 

be 

002 

021 

1 

7 

5 

13 

d 

13 

0 

.206 

0 

.080 

0 

.245 

0 

.171 

0 

.296 

0 

.190 

0 

.000 

0. 

oc 

002 

033 

0 

0 

d 

0 

d 

d 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0. 

oc 

014 

016 

0 

0 

0 

0 

d 

d 

d 

.ddd 

d 

.000 

0 

.000 

0 

-000 

0 

-000 

0 

.000 

0 

.000 

0. 

oe 

014 

021 

0 

0 

0 

0 

0 

0 

d 

.ddd 

d 

.Odd 

0 

.000 

0 

.000 

0 

-000 

0 

.000 

0 

.000 

0. 

oe 

014 

033 

0 

CD 

0 

0 

0 

0 

0 

.000 

d 

.dOd 

d 

.000 

0 

.OdO 

0 

-000 

0 

.000 

0 

.000 

d. 

de 

016 

021 

0 

d 

0 

0 

2 

2 

0 

.017 

0 

.012 

d 

.ddd 

d 

.ddd 

0 

-Odd 

0 

.000 

d 

.017 

d- 

02 

016 

033 

0 

b 

0 

0 

1 

1 

0 

.008 

b 

.2bo 

n 

.000 

0 

.Odd 

d 

-ddd 

d 

.ddd 

0 

.OdS 

0- 

33 

021 

033 

0 

0 

3 

5 

0 

.031 

1 

.000 

0 

.026 

1 

.000 

0 

.047 

1 

.000 

0 

.035 

1 . 

00 

QUESTION  #  :  bQ4 
DATABASE  #   :  154 


♦Accession  Number  Overlap  between  pairs  of  Searchers  * 
************************************ 


Search  Evaluation 


A: 
B: 

e: 

D: 
E: 
F: 


# 
# 


Relevant 


***     Proportions  *** 

si:   F  in  overlap  between  Sl&S^ 


Partially  Relevant  S2:   F  in  overlap  between  S1&S2 


/ 
/ 


for  SI 
for  S2 


#  Not  Relevant 
Total  #  Evaluated 

#  Not  Evaluated 
Total   #  Retrieved 


HI  : 
H2: 
II  : 
12: 


D  for  SI 


D  for  S2 


D  in  overlap  between  S1&S2  / 
D  in  overlap  between  SI &S2  / 

A+B  in  overlap  between  S1&S2  /  A+B  for  Si 

A+B  in  overlap  between  S1&S2  /  A+B  for  S2 

Jl  :   E  in  overlap)  between  S1&S2  /  E  for  SI 

J25   E  in  overlap  between  S1&S2  /  E  for  S2 


Total   for  Question  : 


Searcher 

A 

B 

C 

D 

E 

F 

□09 

6 

1 

3 

•id 

i 

ii 

□IG 

11 

14 

7 

32 

i 

33 

024 

28 

8 

5 

41 

1 

42 

□28 

5 

0 

4 

9 

1 

ID 

□37 

12 

12 

1^ 

34 

2 

36 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

C 

D 

E 

□□9 

□!□ 

□ 

□ 

□ 

□ 

□ 

□□9 

□24 

3 

□ 

2 

5 

Q 

□□9 

□28 

3 

□ 

2 

5 

d 

□09 

□37 

□ 

□ 

d 

d 

d 

□10 

□24 

P 

□ 

□ 

□ 

0 

□10 

□28 

□ 

□ 

D 

d 

□ 

□10 

□37 

2 

1 

□ 

3 

□ 

□24 

□28 

4 

□ 

3 

7 

□ 

□24 

□37 

1 

□ 

□ 

1 

□ 

□28 

□37 

1 

□ 

□ 

1 

□ 

61 


HI 


H^ 


II 


12 


Jl 


□ 
5 
5 

d 
d 

□ 
3 
7 
1 
1 


.□□0 
.454 
.454 

.Odd 

.dOD 
.000 
.09^ 
.166 
.□23 
.  1  □□ 


□□□ 
119 

5dd 
□dd 
ddd 

d-DOD 
0.083 
□  .7^0 
0.027 
0.027 


0.000 
0.500 

d.5dd 
d.ddo 
d .  ddd 

d.ddO 

d.o?3 

0.170 
0 . 024 
0.111 


0.000 
0.121 
d.555 

d .  odd 
d.ddd 
d.ddd 

0.088 
0.777 
0 . 029 
0.029 


0.000 
0.428 
0.428 

d .  ddo 
d.ddd 
d.ood 

d.l20 
0.111 
0.027 
0.200 


0.000 
0.  033 
0.600 

D.ddd 
d .  ddd 
d.ddd 

d.l25 

g.soo 

0.041 
O .  CD41 


0.000  0.1 

□  .□□□  □•! 

□  .□□□  b.i 
d.ddd  □-! 
diddd  d-i 

ddd  □.! 

ddd  d.i 

□□□  □.( 

□□□  □.{ 

□□□  □.( 
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QUESTION  #  :  005 
DATABASE  #   :  148 


*     Accession  Number  Overlap  between  pairs  pi  Searchers  * 


Search  Evaluation 


A:  #  Relevant 

Bs  #  Partially  Relevant 

C:  #  Not  Relevant 

D:  Total  #  Evaluated 

E:  #  Not  Evaluated 

F:  Total  #  Retrieved 


6 

HI 


/  F 


Proportions  *** 
Gl :   F  in  overlap  between  Si&S2 

F  in  overlap  between  S1&S2  / 
D  in  overlap  between  S1&S2  / 

D  in  overlap  between  S1&S2  /  .   

II:   A+B  in  overlap  between  SISSS  /  A+3  for  SI 

12:   A+B  in  overlap  between  Si&S2  /  A+B  for  S2 

Jl :   E  in  overlap  between  Si&S2  /  E  for  SI 

J2:   E  in  overlap  between  S1&S2  /  E  for  S2 


for  SI 
F  for  S2 
D  for  SI 
D  for  S7 


Total   for  Question  : 


Searcher 

A 

B 

C 

D 

E 

F 

□03 

6 

15 

10 

31 

2 

33 

□04 

1 

2 

7 

0 

7 

006 

0 

3 

5 

5 

2 

ID 

013 

b 

0 

g 

0 

5 

5 

026 

1 

0 

3 

4 

0 

4 

Searcher  Overlap 


Searcher 


SI 

S2 

A 

B 

c 

D 

E 

F 

61 

62 

HI 

H2 

11 

12 

Jl 

003 

004 

1 

2 

4 

7 

0 

7 

0 

.212 

1 

.odd 

d 

.225 

1 

.000 

0. 

142 

1 

.000 

0. 

000 

0 

.D\ 

003 

006 

0 

3 

4 

7 

1 

6 

0 

.242 

d 

.add 

d 

.225 

Q 

.575 

0. 

142 

1 

.000 

0. 

500 

b 

.5( 

003 

013 

0 

0 

0 

0 

0 

0 

0 

.POD 

0 

.ggg 

g 

.ggg 

g 

.ogg 

0. 

ggo 

0 

.000 

0. 

000 

0 

.0( 

003 

026 

0 

0 

CD 

0 

0 

0 

0 

.000 

g 

.OOP 

g 

.000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

0 

.oc 

004 

006 

0 

0 

2 

2 

0 

2 

0 

.285 

g 

.2gg 

g 

.285 

D 

.25g 

0. 

000 

0 

.000 

0. 

000 

0 

.Of 

004 

013 

0 

0 

0 

0 

b 

0 

0 

.000 

0 

.000 

0 

.000 

D 

.Qgg 

g. 

000 

0 

.000 

0. 

000 

0 

.0£ 

004 

026 

b 

b 

b 

0 

0 

0 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0. 

ggo 

g 

.000 

0. 

000 

0 

.QC 

006 

013 

d 

b 

b 

b 

b 

0 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0. 

000 

g 

.ggg 

g. 

ggg 

0 

.oc 

006 

026 

0 

b 

0 

d 

b 

0 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0. 

000 

0 

.goo 

0. 

oog 

0 

.DC 

013 

026 

0 

G 

0 

0 

0 

0 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

0 

.DC 
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QUESTION  #  :  006 
DATABASE  #   :  066 

*     Accession  Number  Overlap  between  pairs  of  Searchers  * 
Search  Evaluation  : 


As   #  Relevant 

B:   #  Partially  Relevant 

C:  #  Not  Relevant 

D:   Total  #  Evaluated 

E:   #  Not  Evaluated 

F:   Total   #  Retrieved 


***     Proportions  *** 


SI 
G2 
HI 
H2 
II 
12 
Jl 
J2 


F  in  overlap  between  S1SS2  /  F  for  SI 

F  in  overlap  between  S1SS2  /  F  for  S2 

b  in  overlap  between  S1&S2  /  D  for  SI 

b  in  overlap  between  SiaS2  /  D  for  S2 

A+B  in  overlap  between  S1SS2  /  A+B  for  SI 

A+B  in  overlap  between  Si&S2  /  A+B  for  S2 

E  in  b/erlap  between  S1&S2  /  E  for  SI 

E  in  overlap  between  S1&S2  /  E  for  S2 


Tota^  for  Question  : 

Searcher       A       B       C       D       E  F 


□11 

□ 

□ 

□ 

□ 

i 

1 

□27 

□ 

i 

67 

68 

47 

115 

□38 

□ 

D 

2 

2 

i 

3 

□3? 

2 

39 

45 

20 

65 

□4^ 

□ 

□ 

4 

4 

□ 

4 

Searcher  Overlap 
Searcher 


SI 

S2 

A 

B 

C 

D 

E 

F 

□11 

□27 

□ 

d 

d 

d 

□ 

□ 

oil 

□38 

□ 

□ 

□ 

d 

□ 

□ 

□11 

□39 

□ 

□ 

d 

d 

d 

□ 

□11 

□40 

g 

□ 

d 

d 

d 

0 

□27 

□38 

□ 

□ 

0 

d 

d 

0 

□27 

□39 

□ 

□ 

d 

d 

1 

1 

□27 

□4^ 

□ 

□ 

□ 

d 

□ 

□ 

□38 

□39 

□ 

□ 

0 

□ 

1 

1 

□38 

□4^ 

□ 

□ 

□ 

□ 

□ 

□ 

039 

□4^ 

□ 

□ 

□ 

□ 

□ 

□ 

Ql  G2         HI  H2         11  12  Jl 

□.□□□  □.□□□  □.□□□  □.□□□  □,□□□  □.□□□  □.□□□  □.□( 

□  .□□^  □.□□□  □.□□□  □.□C]^  □,□□□  □.□□□  □.□□□  g.0( 

□.□do  o.odo  0.000  0-000  o.ooo  o.ooo  o.odQ  d-dc 

d.ddd  d.bdo  o.ooo  o.ooo  o.ooo  o.ooo  o.ooo  g.dc 

o.ddd  d.ddd  o.ddd  o.odo  o.ooo  o.ooo  o.ooo  o.oc 

c.C"'s  d.dis  d.ddd  d.ddd  o.ooo  o.ooo  0.021  o.ds 

D.DuJ  o.ddd  d.ddd  d.ddd  d.doo  0.000  0.000  o.oc 

d-333  d.dis  d.ddd  d.ddd  do 000  o.odo  1.000  o.oe 

d-Odo  o.ddd  o.Odd  d.ddd  d.ddd  d.ddd  d.odo  o.oe 

0.000  0.000  0.000  0.000  0.000  0.000  0.000  o.oc 


4ao 

o 
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^UEStiON  #  :  dd7 
DATABASE  #    :  075 


*     Accession  Number  Overlap  between  pairs  of   Searchers  * 


Search  Evaluation  : 


A:  #  Relevant 

B:  #  Partially  Relevant 

C:  #  Not  Relevant  

D:  Total  #  Evaluated 

E:  #  Not  Evaluated 

F:  Total  #  Retrieved 


S2 
Hi 
H2 
ii 
12 
Jl 


Proportions  *** 
F  in  overlap  between  S1&S2  /  F 
F  in  overlap  between  S1SS2  /  F 
D  in  overlap  between  S1&S2  /  D 
D  in  overlap  between  SI &S2  /  D 
A+B  in  overlap  between  S1SS2  / 
A+B  in  overlap  between  Si&S2  / 


for  SI 
for  S2 
for  SI 
for  S2 
A+B  for  SI 
A+B  for  S2 


E  in  overlap  between  S1&S2  /  E  for  SI 
E  in  overlap  between  S1&S2  /  E  for  S2 


Total   for  Question  : 
Searcher       A       B       C  D 


003 

1 

3 

1 

5 

0 

5 

(DG4 

16 

23 

28 

67 

278 

345 

d06 

0 

d 

0 

d 

i 

i 

014 

i 

0 

i 

2 

10 

12 

□26 

3 

1 

i 

5 

13 

18 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

c 

D 

E 

F 

Gl 

62 

HI 

H2 

11 

12 

Jl 

J 

□  03 

004 

0 

0 

0 

0 

0 

0 

0 

-000 

0 

.000 

0 

.000 

b 

.bbb 

0. 

000 

0 

.000 

0. 

000 

0. 

OG 

□  03 

006 

0 

0 

b 

b 

0 

0 

0 

-  000 

0 

.000 

0 

.000 

0 

.oob 

0. 

000 

0 

.000 

0. 

000 

0. 

OG 

□  03 

014 

d 

d 

d 

d 

0 

0 

0 

.000 

□ 

.000 

0 

.000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

0. 

OG 

dd3 

d26 

d 

d 

d 

0 

0 

0 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

0. 

OG 

004 

006 

0 

d 

0 

d 

d 

d 

d 

-ddo 

0 

.000 

0 

.000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

□  . 

OG 

004 

014 

0 

0 

0 

0 

2 

2 

d 

-dds 

0 

.166 

0 

.000 

0 

.000 

0. 

000 

0 

.000 

0. 

007 

0. 

20 

004 

026 

0 

0 

1 

1 

1 

2 

0 

.005 

0 

.111 

d 

.di4 

d 

.2dd 

d. 

dod 

d 

.ddd 

d. 

dd3 

0. 

d7 

006 

□14 

0 

0 

0 

0 

0 

0 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

0. 

00 

006 

026 

0 

0 

0 

0 

0 

0 

0 

.000 

g 

.000 

0 

.000 

a 

.000 

0. 

000 

0 

.000 

0. 

000 

0. 

00 

014 

026 

0 

0 

0 

0 

0 

0 

0 

.000 

0 

.000 

0 

.000 

D 

.000 

0. 

000 

0 

.000 

0. 

000 

0. 

00 

ERIC 


QUESTION  »  :  OOS 
DATABASE* #   :  154 


*     Accession  Number  Overlap  between  pairs  of  Searchers  * 


Search  Evaluation 


A:   #  Relevant 

B:   #  Partially  Relevant 

C:   #  Not  Relevant 

b:   Tot.si  #  Evaluated 
E:   #  Not  Evaluated 
F:   Total  #  Retrieved 


***  Proportions  *** 
Gl :  F  in  overlap  between  Si&S2  / 
G2:  F  in  overlap  between  SidS2  / 
HI :  D  in  overlap  between  SldS2  / 
__H2:  D  in  overlap  between  Si&S2 
II:   A+B  in  overlap  between  S1&L2 


F  for  SI 
F  for  S2 
D  for  Si 
/   D  for  S2 
/  A+B  for  SI 


12:   A+B  in  overlap  between  S1&S2  /  A+B  for  S2 
Jl :  E  in  overlap  between  S1SS2  /  E  for  SI 
J2:   E  in  overlap  between  S1&S2  /  E  for  S2 


Total   for  Question  : 


?her 

A 

B 

C 

D 

E 

F 

□09 

□ 

□ 

a 

6 

g 

_S 

□10 

2 

51 

56 

□ 

56 

□2A 

2 

3 

3^ 

39 

□ 

39 

□28 

0 

□ 

3 

3 

CD 

3 

.037 

□ 

11 

13 

□ 

13 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

c 

D 

E 

F 

Gi 

G2 

Hi 

H2 

ii 

12 

Jl 

J 

□09 

010 

0 

0 

8 

8 

g 

8 

1 

.Odd 

d 

.142 

1 

.OOd 

d- 

142 

0- 

000 

0. 

000 

0. 

000 

0 

.00 

□09 

024 

0 

0 

8 

8 

0 

8 

1 

.ggg 

d 

-205 

1 

.ddd 

0- 

205 

0. 

000 

0. 

000 

0. 

000 

0 

-00 

□09 

026 

0 

b 

0 

0 

0 

0 

0 

-000 

0 

-000 

d 

.ddd 

0- 

ddd 

0- 

000 

0- 

000 

0- 

000 

D 

iOQ 

009 

037 

0 

0 

1 

1 

0 

1 

0 

.125 

0 

-076 

0 

- 1 25 

0. 

076 

0. 

ddd 

0. 

ddd 

d. 

ddd 

d 

.dd 

010 

024 

2 

3 

34 

39 

d 

39 

0 

-696 

1 

.000 

0 

.696 

1  . 

000 

1  . 

000 

1  . 

000 

0- 

ooo 

0 

.00 

bio 

028 

0 

0 

2 

2 

0 

2 

0 

.035 

0 

.666 

0 

.035 

0. 

666 

0. 

000 

0- 

ggg 

0. 

000 

d 

.dd 

dio 

037 

2 

0 

li 

13 

0 

13 

0 

.232 

1 

.000 

0 

.232 

1  . 

000 

0. 

400 

1  - 

000 

0. 

000 

0 

.00 

024 

028 

d 

d 

0 

0 

0 

0 

0 

.000 

0 

.000 

0 

.000 

0. 

000 

0. 

000 

0- 

000 

(D. 

000 

0 

.00 

024 

037 

2 

d 

1 

3 

d 

3 

d 

.076 

d 

.230 

0 

.076 

0. 

230 

0. 

400 

1  - 

000 

0. 

000 

0 

.00 

028 

037 

0 

0 

2 

2 

0 

2 

0 

i666 

0 

.153 

0 

-666 

0- 

153 

0. 

000 

0. 

000 

0. 

000 

0 

-00 

432 
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QUESTION  #  :  009 
DATABASE  #   :  037 


*     Accession  Number  Overlap  between  pairs  ol   Searchers  * 

Search  Evaluation  : 

_   _  ***     Proportions  *** 

fi:   #  Relevant  Gl :   F  in  overlap  between  S1&S2  /  F  for  SI 

B:   #  Partially  Relevant  G2:   F  in  overlap  between  Si&S2  /  F  for  S2: 

C:   #  Not  Relevant  HI:   D  in  overlap  between  Si&S2  /  D  for  SI 

Ds   Total  #  Evaluated         H2:   D  in  overlap  between  S1&S2  /  D  for  S2 

E:   #  Not  Evaluated  II:   A+B  in  overlap  between  SiaS2  /  A+B  for  Si 

F:   Total  #  Retrieved         12:   A+B  in  overlap  between  Si&S2  /  A+B  for  S2 

Jl:   E  in  overlap  between  S1&S2  /  E  for  Si 

J2:   E  in  overlap  between  S1&S2  /  E  for  S2 


Total   for  Question  : 


?her 

A 

B 

C 

D 

E 

F 

002 

1 

6 

_2 

9 

45 

54 

01^ 

2 

6 

2? 

37 

97 

134 

016 

0 

3 

27 

30 

54 

84 

021 

1 

3 

2 

6 

52 

58 

033 

1 

5 

0 

6 

26 

32 

Sea rcher  Oyer i ap 
Searcher 


SI 

S2 

A 

& 

e 

D 

E 

002 

014 

0 

1 

0 

1 

Q 

002 

016 

g 

0 

0 

0 

0 

002 

021 

0 

1 

g 

1 

5 

002 

033 

0 

1 

g 

1 

3 

014 

016 

0 

0 

0 

0 

g 

014 

021 

0 

0 

0 

0 

0 

014 

033 

0 

0 

0 

0 

1 

016 

021 

0 

0 

0 

0 

0 

016 

033 

0 

0 

0 

0 

0 

02:1 

03^ 

0 

0 

0 

0 

2 

Gl  G2         HI  H2          II  12  Jl 


1   O.OIS  0.007  0.111   0.027  0.142  0-125   0.000  g . 00 

G  0.000  b.obo  b.ooo  o.ooo  o.ooo  o.ooo  o.ooo  g.gg 

6  g.lll   0.103  b.iii   0.166  0.142  0.250  0.111  0.09 

4  0.g74  0.125  Q.iil   0.166  0.142  0.166  0.066  0.11 

g  g.goo  o.oog  o.oog  o.bob  b.boo  o.ooo  o.ooo  o.od 

0  g.ggg  g.ggg  g.ggg  o.goo  o.boo  o.boo  o.ooo  o.oo 

1  g.gg7  g.gsi  g.ooo  o-ogg  g.ggg  g.goo  g.oib  0.03, 
g  g.ggg  g.ggg  g.ggg  g.ggg  g.ggg  g.ggg  g.ggg  b.bbi 
o  g.goo  g.goo  g.goo  g.ggg  o.gog  g.ggg  g.goo  g.goi 

2  0.034  0.062  0.000  0-000  0-000  0.000  0.038  0.07< 


4.13 
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QUESTION  #  :  010 
DATABASE  #   :  154 


*     Accession  Number  Overlap  between  pairs  of   Searchers  * 
Search  Evaluation  :  • 


Proportions  *** 

A: 

»  Relevant 

Gl  : 

F  in  overlap  between  SidS2  / 

F 

for 

SI 

Bs 

»  Partially  Relevant 

S2: 

F  in  overlap  between  Si&S2  / 

F 

for 

S2 

C: 

#  Not  Relevant 

HI  : 

D  in  overlap  between  S1&S2  / 

D 

for 

Si 

D: 

Total  #  Evaluated 

H2: 

P  in  overlap  between  Si&S2  / 

D 

for 

S2 

E: 

#  Not  Evaluated 

11  : 

A+B  in  overlap  between  Si&S2 

/ 

A+B 

for 

F: 

Total  #  Retrieved 

12: 

A+B  in  overlap  between  Si&S2 

/ 

A+B 

for 

Jl  : 

E  in  overlap  between  SiaS2  / 

E 

for 

SI 

J2: 

E  in  overlap  between  SI &S2  / 

E 

for 

S2 

Total   for  Question  : 


Searcher       A  B 


009 

1 

g 

010 

0 

0 

024 

2 

0 

028 

0 

0 

037 

2 

1 

C 

D 

E 

F 

0 

1 

3 

4 

2S 

28 

72 

100 

3 

5 

•14 

1? 

4 

4 

IS 

22 

2 

5 

13 

IS 

Searcher 


Si 

S2 

A 

B 

e 

D 

E 

F 

61 

G2 

HI 

H2 

li 

12 

Jl 

00? 

gio 

0 

0 

0 

0 

D 

D 

0 

.000 

0. 

000 

0.000 

0.000 

0 

.000 

0. 

000 

0. 

000 

0. 

0 

009 

024 

1 

0 

0 

1 

2 

3 

0 

.750 

0. 

157 

i  .Odd 

d.2dd 

i 

.000 

0. 

500 

0. 

666 

Q- 

1 

□09 

028 

0 

0 

0 

0 

0 

P 

0 

.ODD 

D. 

ODD 

D.OOD 

D.DDO 

0 

.000 

0. 

000 

0. 

000 

0. 

0 

009 

037 

1 

0 

0 

1 

1 

2 

0 

.500 

0. 

111 

1  .000 

0.200 

1 

.god 

D. 

333 

0. 

333 

d. 

0 

010 

024 

0 

0 

0 

0 

0 

0 

Q 

.ODD 

0. 

ODD 

O.OOQ 

g.ooD 

D 

.000 

0. 

000 

0. 

000 

0. 

0 

Old 

028 

0 

0 

0 

0 

0 

0 

0 

.ODD 

0. 

000 

g.ggg 

g.ggg 

D 

.ODD 

d. 

ggo 

0. 

000 

0. 

0 

dio 

037 

b 

0 

0 

0 

1 

1 

0 

.  01 0 

0. 

055 

0.000 

g.ggg 

g 

.ggg 

0. 

000 

g. 

013 

0. 

0 

024 

028 

0 

0 

0 

0 

0 

0 

0 

.000 

0. 

000 

0.000 

0.000 

0 

.000 

0. 

000 

0. 

000 

0. 

Ol 

024 

037 

2 

d 

i 

3 

6 

9 

0 

.473 

0- 

500 

0.600 

0.600 

1 

.000 

0. 

666 

0. 

428 

0. 

4. 

028 

037 

0 

0 

0 

0 

0 

0 

0 

.000 

0. 

000 

0-000 

0.000 

0 

.000 

0. 

000 

0. 

000 

0. 

0( 

4Bi 
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QUESTiON  #  5  Oil 
DATABASE  #   :  154 


*     Accession  Number  Overlap  between  pairs  of   Searchers  * 
Search  Evaiuation  : 


_    ***  _Prdpcrtiphs  *** 

A:   #  Relevant    Gl :  F  in  overlap  between  SI &S2  /  F  for  SI 

B:   #  Partially  Relevant  G2:  F  in  overlap  between  S1&S2  /  F  for  S2 

e:  #  Not  Relevant   His  D  in  overlap  between  S1&S2  /  D  for  Si 

ps   Total  #  Evaluated         H2:  D  in  overlap  between  S1SS2  /   D  for  S2 

E:   #  Not  Evaluated  ii :  A+B  in  overlap  between  S1&S2  /  A+B  for  Si 

F:   Total  #  Retrieved         12:  A+B  in  overlap  between  S1SS2  /  A+B  for  S2 

Ji:  E  in  overlap  between  S1SS2  /  E  for  Si 

J2:  E  in  overlap  between  S1&S2  /  £  for  S2 


Total   for  Question  : 


Searcher       A  B 


□d9 

i 

□ 

dio 

3 

6 

□24 

2 

6 

028 

2 

6 

037 

□ 

1 

C 

D 

E 

F 

1 

2 

7 

9 

31 

40 

58 

98 

is 

23 

22 

45 

i9 

27 

27 

54 

1 

2 

3 

5 

Searcher  Overlap 


Searcher 


SI 

S2 

A 

B 

C 

D 

E 

F 

Gl 

62 

Hi 

H2 

009 

□!□ 

□ 

□ 

i 

1 

7 

8 

□  - 

888 

0.081 

0.500 

g. 

025 

009 

024 

□ 

□ 

b 

□ 

2 

□  . 

222 

0.044 

o.oog 

g. 

ODD 

009 

028 

0 

.  □ 

□ 

□ 

2 

2 

□  - 

222 

0 . 037 

g.goo 

g. 

000 

009 

037 

□ 

□ 

i 

i 

3 

4 

0- 

444 

0 . 800 

0.500 

0. 

500 

010 

□24 

2 

5 

11 

i8  - 

2^ 

38 

□  - 

387 

0.844 

0.450 

0. 

782 

010 

□28 

2 

5 

14 

2i 

25 

46 

□  . 

469 

0.851 

0.52S 

Q. 

777 

010 

□37 

0 

□ 

1 

1 

3 

4 

□  • 

□4^ 

0.800 

0.025 

0. 

500 

024 

□28 

2 

6 

15 

23 

22 

45 

i  - 

ddo 

0.833 

i  .  000 

0. 

851 

024 

□37 

□ 

□ 

□ 

□ 

□ 

□ 

□  . 

dDd 

d.ddd 

b.doo 

0. 

000 

□28 

□37 

□ 

□ 

□ 

□ 

□ 

□ 

□  . 

000 

0.000 

0.000 

0. 

000 

II 


12 


Jl 


O.DDD 
D.DdD 
D.DDD 
D-DDD 
D.777 
g.777 

0.  00g 

1 .  oog 
o.odo 

O  .  000 


0.000 

d.doG 
d.dSG 

OOD 

ggg 

000 
000 
000 


1 .000  0.1 
0.285  0.0 
di285  0.0 
D .  428  i  .  01 
0.344 
D-431 

0.  D51 

1.  DDD  D-8-: 
0.000  0.0( 
0.000  0.0( 


D.91 
0.9: 
1 .0( 
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QUESTION  #   :  012 
DATABASE  #   :   013      *  • 

*     "^PPfffipn  Number  Overlap  Oetween  pairs  of  Searchers  * 
Search  Evaluation  : 


Proportions  *** 

A: 

#  Relevant 

m : 

F  in  overlap  between 

SiaS2  / 

F 

for 

SI 

B: 

#  Partially  Relevant 

S2: 

F  in  overlap  between 

SlfiS2  / 

F 

for 

S2 

C: 

#  Not  Relevant 

HI  : 

D  in  overlap  between 

S1&S2  / 

D 

for 

SI 

D: 

Total  #  Evaluated 

H2: 

D  in  overlap  between 

S1&S2  / 

D 

for 

S2 

E: 

»  Not  Evaluated 

11  : 

A+B  i ri  over 1 ap  betwee 

?n  S1&S2 

/ 

A+B 

for 

F: 

Total  #  Retrieved 

12: 

A+B  in  overlap  between  S1&S2 

/ 

A+B 

for 

Jl  : 

E  in  overlap  between 

SldS2  / 

E 

for 

SI 

J2: 

E  in  overlap  between 

S1&S2  / 

E 

for 

S2 

Total   for  Question  s 


Searcher       A  B 


□05 

0 

3 

gii 

9 

□27 

1 

4 

029 

0 

4 

040 

1 

1 

e 

D 

E 

F 

30 

33 

17 

50 

31 

44 

38 

82 

7 

12 

7 

19 

5 

9 

4 

13 

2 

4 

3 

7 

Searcher  Overlap  : 


Searcher 


Si 

32 

A 

B 

e 

b 

E 

F 

Gl 

G2 

HI 

H2 

11 

12 

Jl 

w 

005 

Oil 

0 

0 

2 

2 

4 

6 

0 

.120 

0. 

073 

0 

.060 

0 

.045 

0.000 

0 

.000 

0 

.235 

O.IC 

005 

027 

0 

0 

0 

0 

d 

d 

d 

.ddd 

0. 

000 

0 

.000 

0 

.  000 

o.ood 

0 

.odd 

0 

.000 

o.oc 

005 

029 

0 

0 

0 

0 

0 

0 

0 

.000 

0. 

000 

0 

.000 

0 

.000 

O.GOO 

0 

.000 

0 

.000 

o.oc 

005 

040 

0 

0 

0 

D 

0 

0 

0 

.000 

d. 

000 

d 

.ood 

0 

.000 

0.000 

0 

.000 

0 

.000 

o.oc 

Oil 

027 

0 

4 

3 

7 

4 

11 

0 

.134 

G. 

578 

0 

.159 

0 

.583 

0.307 

0 

.800 

0 

.105 

0.57 

Oil 

CD29 

0 

4 

2 

6 

3 

9 

0 

.109 

0. 

692 

0 

.136 

0 

.666 

0.307 

1 

.000 

0 

.078 

0.7B 

Oil 

CD40 

1 

1 

1 

3 

0 

3 

0 

.036 

0. 

428 

0 

.068 

0 

.750 

P. 153 

1 

.Pod 

0 

.000 

Q.oe 

027 

029 

0 

4 

5 

9 

4 

13 

0 

.684 

1  . 

POP 

P 

.750 

1 

.OOP 

P .  SPP 

1 

.PPP 

p 

.571 

•1  .eo 

027 

040 

0 

0 

0 

0 

0 

0 

0 

.000 

0. 

000 

P 

.PPP 

P 

.PPP 

P .  000 

P 

.000 

p 

.000 

Q.OO 

029 

040 

0 

0 

0 

0 

0 

0 

0 

.000 

0. 

000 

0 

.000 

0 

.000 

0.000 

0 

.000 

0 

.000 

0.00 

QUESTION  #  :  D13 
DATABASE  #   :  015 


*     Accession  Number  Oyeriap  between  pairs  of  Searchers  * 


Search 'Evaluation 


*** 

A :  #  Re 1 evant  SI : 

B:  #  Partially  Relevant  G2: 
C:   #  Not  Relevant  HI: 
D:  Total  #  Evaluated 
E:  #  Not  Evaluated 
F:   Total  #  Retrieved 


Proportions  *** 
F  in  overlap  between  Si&S2  /  F  for  Si 
F  in  overlap  between  Si&S2  /  F  for  S2 
P  in  overlap  between  Si&S2  /   D  for  Si 
C>  in  overlap  between  Si&S2  /   D  for  S2 
II:   A+B  in  overlap  between  Si&S2  /  A+B  for  Si 
12:   A+B  in  overlap  between  Si&S2  /  A+B  for  S2 
Jl :   E  in  overlap  between  S1&S2  /  E  for  Si 
J2:   E  in  overlap  between  S1&S2  /  E  for  S2 


H, 


Total   for  Question 


Searcher 

A 

B 

C 

D 

£ 

F 

003 

3 

5 

22 

30 

3 

33 

00^ 

1 

1 

0 

2 

0 

006 

2 

8 

19 

29 

1 

30 

013 

3 

IS 

39 

60 

5 

65 

026 

14 

37 

55- 

7 

62 

Searcher  Overlap 


Searcher 


SI 

S2 

A 

B 

e 

D 

E 

F 

61 

Q2 

HI 

H2 

11 

12 

Jl 

003 

004 

1 

0 

0 

1 

0 

1 

0-030 

d. 

500 

0.033 

0. 

500 

0. 

125 

0. 

500 

0. 

000 

003 

006 

2 

1 

3 

6 

d 

6 

0.181 

di 

20d 

d.2od 

d. 

206 

0. 

375 

0. 

300 

0. 

000 

003 

013 

2 

1 

2 

5 

0 

5 

0.151 

0. 

076 

D.166 

0. 

083 

d. 

375 

d. 

142 

d. 

dob 

003 

026 

2 

2 

3 

7 

0 

7 

0.212 

0. 

112 

0.233 

0. 

127 

d. 

5Dd 

d. 

222 

d. 

Odd 

004 

006 

1 

0 

0 

1 

0 

1 

0.500 

0. 

033 

0.500 

0. 

034 

0. 

500 

0. 

idd 

d. 

ODD 

004 

013 

1 

i 

0 

2 

0 

2 

1  . 000 

0. 

030 

1  .000 

0. 

033 

1 . 

000 

0. 

095 

d. 

000 

004 

026 

1 

1 

0 

2 

d 

2 

1  . 000 

0. 

032 

1  .000 

0. 

036 

1 . 

000 

d- 

111 

0. 

Odd 

006 

013 

1 

3 

2 

6 

b 

6 

0 . 200 

0. 

092 

0 . 206 

0. 

100 

0. 

400 

0. 

190 

0. 

ood 

006 

026 

1 

4 

6 

13 

i 

14 

0.466 

0. 

225 

0.448 

0. 

236 

0. 

500 

0. 

277 

1 . 

000 

013 

026 

3 

7 

7 

17 

19 

0.292 

0. 

306 

0.283 

0. 

309 

0. 

476 

0. 

555 

0. 

400 

D.Gt 


ERIC 


4^7 


QUESTION  #  :  014 
DATABASE  #   :  151 


********************** 

*  Accession  Number  Overlap  between  pairs  of  Searchers  * 
*************************************^t^t^^^^t^(.^(.^(.^i.^(.^^t^t^(.^t^(.^t^^(.^^ 

Search  Evaluation  : 


*** 

Prdpbrtiphs  *** 

A: 

#  Relevant 

Gl : 

F  in  overlap  between  S15S2  / 

F 

for 

SI 

B: 

#  Partially  Relevant 

G2s 

F  in  overlap  between  S1SS2  / 

F 

for 

S2 

0: 

#  Not  Relevant 

HI : 

b  in  overlap  between  S1SS2  / 

D 

for 

SI 

D: 

Total  #  Evaluated 

H2: 

D  in  overlap  between  S1SS2  / 

D 

for 

S2 

E: 

#  Not  Evaluated 

1±  : 

A+B  in  overlap  between  Sias2 

/ 

A+B 

for 

F: 

Total  #  Retrieved 

i2: 

A+B  in  overlap  between  SiaS2 

/ 

A+B 

for 

Ji  : 

E  in  overlap  between  Si&S2  / 

E 

for 

SI 

J2: 

E  in  overlap  between  S1&S2  / 

E 

for 

S2 

Total   for  Question  : 


Searcher       A  B 


003 

16 

26 

004 

10 

15 

006 

11 

19 

026 

0 

1 

033 

24 

50 

C 

D 

E 

F 

9 

51 

1 

52 

5 

30 

1 

31 

7 

37 

2 

39 

32 

33 

0 

33 

14 

88 

1 

89 

Searcher  Overlap 
Searcher 


SI 

S2 

A 

B 

c 

D 

E 

F 

Gl 

G2 

HI 

fl2 

II 

12 

JI 

J 

003 

004 

10 

15 

4 

29 

1 

30 

0.576 

0. 

967 

0 

.568 

0.966 

0.595 

1  .000 

1  .GOD 

1.00 

003 

006 

10 

15 

4 

29 

1 

30 

0. 

576 

0. 

769 

0 

.568 

0.783 

0.595 

0.833 

1  ,ogo 

0.50 

003 

026 

0 

0 

1 

1 

0 

1 

0. 

019 

0. 

030 

0 

.019 

0  .030 

0.000 

0.000 

D.OOO 

q.DQ 

003 

033 

7 

13 

3 

23 

1 

24 

0. 

461 

0. 

269 

0 

.450 

0.261 

0.476 

0.270 

1  . 000 

1  •  00 

004 

006 

Id 

15 

5 

30 

1 

31 

1  . 

000 

0. 

794 

1 

.000 

O.Slb 

1 . 000 

0 . 833 

1  . 000 

0.501 

004 

026 

d 

d 

2 

2 

0 

2 

0. 

064 

0. 

060 

0.066 

0.060 

0.000 

0.000 

0.000 

0.001 

004 

033 

5 

ii 

4 

2d 

1 

21 

d. 

677 

0. 

235 

0 

.666 

0.227 

0.640 

0.216 

1  . 000 

1  -  00( 

D06 

026 

0 

0 

2 

2 

0 

2 

0. 

051 

0. 

060 

0 

.054 

0.060 

0.000 

0.000 

0.000 

0.001 

006 

033 

5 

13 

5 

23 

1 

24 

0. 

615 

a. 

269 

0 

.621 

0.261 

0.600 

0^243 

0.500 

1 .00( 

026 

033 

0 

0 

1 

1 

0 

1 

0. 

030 

0, 

Oil 

0 

.030 

0.011 

0.000 

0.000 

0.000 

0.00( 
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QUESTION  #  s  015 
DATABASE  #   :  154 


*     Accession  Number  Overlap  between  pairs  of  Searchers  * 
Search  Evaluation  : 


***     Proportions  *** 

A:   #  Relevant   Ql :   F  in  overlap  between  S1&S2  /  F  for  SI 

B:   #  Partially  Relevant  62:   F  in  overlap  between  S1&S2  /  F  for  S2 

C:   #  Not  Relevant    HI:   D  in  overlap  between  S1&S2  /  D  for  SI 

D:   Total   #  Evaluated         H2:   D  in  over lap  between  S1&S2  /  D  for  S2 
E:   #  Not  Evaluated  ii :   A+B  in  over lap  between  si &S2  /  A+B  for  SI 

F:   Total   #  Retriiieved         12:   A+B  in  overlap  between  Si&B2  /  A+B  for  S2 

Jl :  E  in  overlap  between  Si&S2  /  E  for  Si 
J2:   E  in  overlap  between  Si&S2  /  E  for  S2 


Total   for  Question  : 


Searcher 

A 

B 

C 

D 

E 

F 

□09 

9 

12 

3 

24 

36 

60 

□IG 

27 

61 

3d 

118 

144 

262 

Q24 

14 

25 

5 

44 

63 

107 

D28 

22 

41 

12 

75 

1Q5 

18Q 

D37 

13 

21 

9 

43 

52 

95 

Searcher  Overlap 


Searcher 

SI 

S2 

A 

B 

C 

D 

E 

F 

Q2 

HI 

H2 

11 

12 

Jl 

□09 

dio 

9 

12 

3 

24 

36 

60 

1  . 

000 

0 

.229 

1 

.000 

0 

.203 

1  . 

000 

0 

.233 

1 

.000 

dd9 

024 

i 

4 

0 

5 

17 

22 

0. 

366 

0 

.205 

0 

-208 

CD 

.113 

0. 

238 

0 

.128 

0 

•  472 

GG9 

Q28 

9 

11 

3 

23 

36 

59 

d. 

983 

0 

.327 

0 

.958 

0 

.  306 

0. 

952 

0 

.  31 7 

1 

.  DOO 

009 

037 

9 

9 

3 

21 

26 

47 

Oi 

783 

0 

.494 

0.875 

G 

.488 

0. 

v^.57 

0 

.529 

0 

.722 

Old 

024 

14 

25 

5 

44 

63 

1G7 

Oi 

4G8 

1 

.000 

0 

.372 

1 

.000 

0. 

443 

1 

.000 

0 

.437 

010 

02S 

22 

41 

12 

75 

105 

180 

0. 

687 

1 

iddG 

G 

i  635 

i 

.000 

0. 

715 

i 

.000 

0 

.729 

010 

037 

13 

21 

9 

43 

52 

95 

0. 

362 

1 

.GGO 

P 

.364 

1 

.000 

G. 

386 

1 

.000 

0 

.361 

024 

028 

12 

24 

4 

40 

62 

102 

P. 

953 

0 

.566 

P 

.909 

P 

.  533 

P. 

923 

G 

.571 

0 

.984 

024 

037 

3 

13 

2 

IS 

33 

51 

p. 

476 

0 

.536 

0 

.409 

0 

.413 

P. 

410 

0 

.47P 

d 

.523 

028 

037 

13 

21 

9 

43 

52 

95 

0. 

527 

1 

.000 

0 

.573 

1 

.000 

0. 

539 

1 

.000 

0 

.495 

g.i 

0.3 
O.E 

•1  .c 
•1  .c 
■1  .e 


4  m 

o 

ERIC 


QUESTION  #  :  016 
DATABASE  #   :  Oil 


*     Accession  Number  Overlap  between  pairs  of  Searchers  * 
Search  Evaluation  3 


***     Proportions  *** 


As 

#  Relevant 

Gl  : 

F  in  overlap  between  Si&S2  / 

F 

for 

SI 

B: 

#  Partial Relevant 

S2: 

F  in  overlap  between  Si&S2  / 

F 

for 

S2 

Cs 

#  Not  Rel  ^>^'aht 

HI : 

p  in  overlap  between  Si&S2  / 

P 

for 

SI 

D: 

Total  #  Evaluated 

H2: 

P  in  overlap  between  SI &S2  / 

D 

for 

S2 

E: 

#  Not  Evaluated 

11  : 

A+B  i h  over 1 a p  between  Si  &S2 

/ 

A+B 

for 

SI 

Fs 

Total  #  Retrieved 

12: 

A+B  in  overlafD  between  S1&S2 

/ 

A+B 

for 

S2 

Jl  : 

E  in  overlap  between  S1SS2  / 

E 

for 

SI 

J2: 

E  in  overlap  between  S1SS2  / 

E 

for 

S2 

Total   for  Question  s 


Searcher 

A 

B 

C 

D 

E 

F 

□02 

20 

21 

30 

71 

g 

71 

□05 

2 

3 

3 

8 

□ 

5 

□13 

4 

4 

37 

45 

□ 

45 

□16 

15 

17 

27 

59 

□ 

59 

□21 

3 

4 

20 

27 

□ 

27 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

C 

D 

E 

F 

Gi 

62 

Hi 

H2 

11 

12 

Jl 

J 

□□2 

□□5 

1 

g 

g 

1 

g 

1 

0 

.014 

0 

.125 

0 

.014 

0 

.125 

g 

.024 

b 

.2Qb 

b. 

obo 

0 

.00 

0^2 

□13 

3 

3 

_  s 

14 

g 

14 

g 

.197 

g 

.311 

0 

.197 

g 

.311 

b 

.146 

b 

.750 

0. 

000 

0 

.00 

□□2 

□16 

13 

13 

15 

41 

0 

41 

0 

.577 

g 

.694 

g 

.577 

0 

.694 

0 

.634 

0 

.812 

0. 

000 

Q 

.00 

□□2 

□21 

3 

3 

0 

6 

0 

6 

0 

.084 

0 

.  222 

g 

.gs4 

D 

.  222 

g 

.146 

0 

.857 

0. 

ggo 

b 

.00 

DOS 

□13 

0 

0 

0 

0 

0 

0 

0 

.000 

0 

.000 

0 

.000 

g 

.goo 

g 

.ggg 

0 

.ogo 

0. 

000 

0 

.00 

DDS 

□16 

1 

0 

0  • 

1 

0 

1 

0 

.125 

0 

.016 

0 

.125 

0 

.016 

0 

.200 

0 

.031 

g. 

ggg 

g 

.gg 

DBS 

021 

b 

0 

0 

0 

0 

0 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

g 

.ogg 

g. 

ggg 

□ 

.go 

□13 

016 

3 

1 

4 

3 

b 

3 

0 

.177 

0 

.135 

0 

.177 

0 

.135 

0 

.500 

0 

.125 

0. 

000 

g 

.gg 

□13 

021 

0 

i 

2 

3 

0 

3 

0 

.066 

0 

.ill 

0 

.066 

0 

.111 

0 

.125 

0 

.142 

0. 

000 

0 

.00 

□16 

021 

1 

3 

0 

4 

0 

4 

0 

.067 

0 

.148 

0 

.067 

0 

.148 

0 

.125 

0 

.571 

0. 

000 

0 

.001 

4W 

ERIC 


QUESTION  #  :  017 
DATABASE  #   :  005 


*     Accession  Number  Overlap  between  pairs  of  Searchers  * 
Search  Evaluation  : 


***     Proportions  *** 
A:   #  Relevant  iSl::i  F  in  overlap  between  /  F  for  SI 

Bs  #  Partially  Relevant  G2:  F  in  overlap  between  SidS2  /  F  for  S2 
Cs   #  Not  Relevant  HI:   D  in  overlap  between  S1&S2  /   D  for  SI 

D:   Total  #  Evaluated         H2:   D  in  overlap  between  S1&S2  /   D  for  S2 
E:   #  Not  Evaluated  II:   A+B  in  overlap  between  S1&S2  /  A+B  for  SI 

F:   Total  #  Retrieved         12:   A+B  in  overlap  between  SI §S2  /  A+B  for  S2 

Jl :  E  in  overlap  between  S15S2  /  E  for  SI 
J2:   E  in  overlap  between  S15S2  /  E  for  S2 


Total   for  Questi 


Searcher       A  B 


Oil 

6 

9 

027 

4 

2 

03d 

0 

1 

039 

9 

040 

7 

6 

C        D       E  F 


38  53  64  117 

IS  24  135  159 

O  1  0  1 

8  39  94  133 

19  32  75  107 


Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

c 

D 

E 

F 

Gl 

62 

HI 

H2 

11 

12 

Jl 

« 

01  i 

027 

2 

0 

0 

2 

5 

7 

d- 

d5? 

d- 

044 

0 

.037 

0 

.083 

0. 

1 33 

d .  333 

d. 

078 

0 

.0^ 

Oil 

038 

0 

d 

0 

0 

0 

0 

0. 

000 

Oa 

000 

0 

.000 

0 

.000 

0. 

000 

0.000 

0. 

000 

0 

.0£ 

Oil 

059 

1 

1 

0 

2 

6 

6 

0- 

066 

0. 

060 

0 

.037 

0 

.051 

0. 

1 33 

0.064 

0. 

093 

0 

.0^ 

Oil 

040 

1 

b 

0 

1 

2 

3 

CD. 

025 

0. 

028 

0 

.018 

d 

.031 

0. 

066 

0,076 

0. 

031 

0 

.0^ 

027 

038 

d 

d 

0 

0 

0 

0 

0. 

000 

0- 

000 

0 

.000 

0 

.000 

g. 

OOP 

0.000 

0. 

DOO 

P 

.0€ 

027 

039 

2 

d 

d 

10 

12 

0. 

075 

0- 

090 

0 

.083 

0 

.051 

0. 

333 

0.064 

p. 

074 

0 

.IC 

027 

040 

1 

0 

d 

1 

4 

5 

0. 

031 

0 

046 

0 

.041 

0 

.031 

0. 

166 

0.076 

0. 

029 

P 

.DE 

038 

039 

0 

0 

d 

d 

d 

0 

d. 

doo 

G. 

000 

0 

.000 

0 

.000 

0. 

000 

0.000 

0. 

000 

0 

-DC 

U38 

040 

P 

1 

0 

1 

0 

i 

1  . 

dod 

d. 

009 

1 

.000 

0 

.  031 

1 . 

000 

0.076 

0. 

000 

0 

.OC 

039 

040 

4 

0 

2 

6 

13 

19 

0. 

142 

0. 

177 

0 

.153 

0 

.187 

0. 

129 

0.307 

0. 

138 

0 

.17 
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QUESTION  #  :  018 
DATABASE  #   :  OlS 


*     Accession  Number  Overlap  between  pairs  of  Searchers  * 


Search  Eva 1 uat ion 


A:   #  Relevant 

B:  #  Partially  Relevant 

C:   #  Not  Relevant 

D:  Total  #  Evaluated 

E:   #  Not  Evaluated  

F:   Total  #  Retrieved 


Total   for  Question  : 


***  _Prc?pbrtipns  *** 


Searcher  Overlap 


Gl :   F  in  overlap  between  Sl&S 
G2: 
HI  : 
H2: 


Searcher 

A 

B 

C 

D 

E 

F 

□03 

0 

0 

0 

0 

1 

1 

004 

55 

29 

115 

541 

656 

006 

IS 

ii 

10 

39 

30 

69 

014 

0 

d 

d 

d 

i 

1 

026 

0 

0 

0 

0 

1 

1 

F 
F 


for  SI 
for  S2 


2  / 

F  in  overlap  between  SlStS2  / 
D  in  overlap  between  S1SS2  /   p  for  SI 
P  in  overlap  between  S15S2_/  D  for  S2 


il :  A+B  in  overlap  between  S15S2 
i2:  A+B  in  overlap  between  S1SS2 
Jl :   E  in  overlap  between  S1SS2  / 


/  A-»-B  for  SI 
/  A+B  for  S2 
E  for  SI 


J2:   E  in  overlap  between  S1&S2  /  E  for  S:: 


Searcher 


SI 

S2 

A 

B 

c 

D 

E 

F 

Gl 

G2 

HI 

R2 

11 

12 

Ji 

003 

dd4 

d 

d 

0 

0 

0 

0 

0 

.  000 

0 

.000 

0. 

000 

0. 

000 

d- 

ddd 

d. 

ddO 

D. 

ood 

003 

d06 

d 

0 

0 

0 

0 

0 

0 

.000 

0 

.000 

d. 

000 

0. 

000 

d. 

ddd 

d- 

ddO 

0. 

Odd 

003 

di4 

d 

0 

d 

0 

i 

1 

1 

.000 

1 

-000 

0- 

000 

0. 

000 

0. 

000 

d- 

Odd 

1 . 

ddd 

dd3 

d26 

d 

d 

3 

0 

1 

1 

i 

.000 

1 

.000 

0. 

000 

0. 

OCDO 

0. 

000 

0. 

000 

1 . 

000 

0d4 

006 

13 

6 

4 

23 

15 

38 

e 

.057 

0 

.550 

0. 

200 

0. 

589 

0. 

226 

0. 

655 

c. 

027 

0d4 

014 

d 

d 

d 

d 

d 

0 

0 

.000 

0 

.000 

0. 

000 

0- 

000 

0- 

000 

0. 

000 

0. 

000 

004 

026 

d 

d 

0 

d 

d 

d 

d 

.odd 

6 

.odd 

d. 

odd 

^. 

000 

□  . 

000 

0. 

000 

0. 

000 

006 

014 

0 

0 

d 

d 

0 

d 

0 

.000 

0 

.000 

0. 

000 

0. 

000 

0. 

000 

0. 

000 

0. 

000 

006 

026 

0 

0 

d 

d 

0 

0 

0 

.000 

0 

.000 

0. 

000 

0. 

000 

0. 

000 

0. 

000 

0. 

000 

014 

026 

6 

0 

0 

0 

1 

1 

1 

.000 

1 

.000 

0. 

000 

0. 

000 

0. 

000 

0. 

000 

1 . 

000 

•i.Oi 

0.51 


□  .□( 


ERIC 


QUESTION  #  :  019 
DATABASE  #   :  d75 


*     Accession  Number  OverlajD  between  pairs  of  Searchers  * 
Search  Eva 1 uat ion  : 


Proportions  •>i-** 

A: 

#  Relevant 

Gi: 

F  in  overlap  between  SiaS2  / 

F 

for 

SI 

B: 

#  Partial ly  Relevant 

62: 

F  in  overlap  between  S1&S2  / 

F 

for 

S2 

C: 

#  Not  Relevant 

HI  : 

D  in  overlap  between  Si  aS2  / 

D 

for 

SI 

D: 

total  #  Evaluated 

H2: 

D  in  overlap  between  S1&S2  / 

D 

for 

S2 

E: 

#  Not  Evaluated 

11 : 

A+B  in  overlap  between  S1&S2 

/ 

A+B 

for 

SI 

F: 

Total   #  Retrieved 

12: 

A+B  in  overlap  between  Si&S2 

/ 

A+B 

for 

S2 

Jl : 

E  in  overlap  between  Si&S2  / 

E 

for 

Si 

J2: 

E  in  overlap  between  S1&S2  / 

E 

for 

S2 

Total   for  Question  : 


Searcher       A  B 


ggi 

0 

g 

005 

4 

5 

008 

6 

15 

013 

6 

12 

017 

7 

14 

C 

D 

E 

F 

1 

1 

1 

2 

3 

12 

1 

13 

15 

36 

40 

2 

20 

5 

25 

22 

43 

IS 

61 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

e 

D 

E 

F 

Si 

001 

005 

g 

0 

0 

g 

g 

S 

d.ddd 

001 

005 

0 

g 

0 

g 

g 

0 

0.000 

001 

013 

0 

0 

0 

0 

0 

g 

g.ggo 

001 

017 

0 

0 

1 

1 

1 

2 

1 .  ggg 

005 

008 

2 

2 

0 

4 

0 

4 

g.307 

005 

013 

2 

3 

1 

6 

0 

6 

0.461 

005 

017 

i 

d 

0 

1 

0 

1 

0.076 

dds 

013 

4 

4 

D 

6 

1 

9 

0  -  225 

dos 

017  . 

2 

5 

1 

s 

1 

9 

0 . 225 

013 

017 

2 

0 

0 

2 

0 

2 

O.OSO 

^2         HI  H2          II  12  Jl 

d.ddd. 0.000  o.ooo  o.ooo  o.ooo  o.ooo  g.o( 
Q.dgo  o.odd  o.ooo  o.ooo  o.ooo  O-'doo  o.oc 
g.ggg  o.goo  d.doo  o.ooo  o.ooo  o.ooo  o.oc 

D.032  l.ogg  g.023  d.OOO  o.ooo  i.OOO  0.05 

0.100  g.333  0.111  g.444  0.190  o.ggd  o.oc 
g.240  0.500  g.sgg  g.555  g.277  g.doo  o.de 
g.gi6  g.gss  g.023  0.111  g.g47  g.oog  g.oe 
0.360  g.222  g.4go  g.3SD  0.444  o.ssg  g.2G 
0.147  0.222  g.is6  g.333  g.333  g.25g  g.gE 

0.032  0.100  0.046  0.111   0.095   0.000  O.OC 


443 


QtilSTiON  #  :  020 
DATABASE  #    :  015 


•K- -K- -K- -K- -K- -K- -K- -K- -K- « 

*     Acbessibri  Number  Overlap  between  pairs  of  Searchers  * 


Search  Evaluation 


A:  #  Relevant 

B:  #  Partial ly  Relevant 

C:  #  Not  Relevant  

D:  Total  #  Evaluated 

E:  #  Not  Evaluated  

F:  Total  #  Retrieved 


Proportions 

F  in  overlap  between  S1SS2  /  F  for  SI 

F"  in  overlap  between  S1SS2  /  F"  for  S2 

D  in  overlap  between  S1SS2  /  D  for  SI 

D  in  overlap  between  SI SS2  /  D  for  S2 

Tl:   A+B  in  overlap  between  S1&S2  /  A+B  for 

i2:   A+B  in  over lap  between  S1&S2  /  A+B  for 

Ji  :   E  in  overlap  between  Si&S2  /  for  Si 

J2:   E  in  overlap  between  S1&S2  /  E  for  S2 


Gl  : 
G2: 
HI  : 
H2: 


SI 
S2 


Total   for  Question  : 


Searchher       A       B       C       D       E  F 


001 

0 

1 

1 

2 

2 

4 

oos 

i 

0 

3 

4 

5 

9 

013 

B 

'  11 

33 

49 

43 

92 

017 

0 

4 

10 

14 

25 

39 

032 

2 

2 

6 

10 

8 

IS 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

c 

D 

E 

F 

Gl 

G2 

HI 

H2 

11 

12 

Jl 

001 

005 

0 

0 

0 

0 

0 

0 

0. 

000 

0. 

000 

0. 

000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

0 

.□c 

001 

013 

0 

0 

0 

0 

0 

0 

0. 

000 

0- 

000 

0- 

000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

0 

.oc 

ddi 

017 

0 

d 

d 

d 

d 

0 

0. 

000 

0. 

000 

0. 

000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

0 

.□c 

ddi 

d32 

d 

d 

Q 

d 

d 

0 

0. 

000 

0. 

000 

0. 

000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

0 

.□£ 

005 

013 

d 

Q 

0 

d 

1 

1 

0. 

111 

0- 

010 

0- 

000 

0 

.000 

0. 

000 

0 

.000 

0. 

2dd 

d 

.OS 

005 

017 

0 

0 

0 

d 

d 

d 

0. 

000 

Oi 

000 

0- 

000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

0 

.□E 

005 

032 

0 

0 

0 

0 

0 

0 

0. 

000 

0. 

000 

0, 

000 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

0 

.□c 

013 

017 

0 

2 

3 

5 

3 

s 

g. 

086 

D. 

205 

0. 

102 

0 

.357 

0. 

125 

0 

.500 

0. 

D69 

0 

.12 

013 

032 

2 

3 

7 

3 

10 

0. 

Ids 

0. 

555 

0. 

142 

0 

.700 

0. 

250 

1 

.000 

0. 

069 

D 

.37 

017 

032 

0 

1 

2 

3 

2 

5 

0. 

128 

0. 

277 

0. 

214 

0 

.300 

0. 

250 

0 

.250 

0. 

080 

0 

.25 

ERIC 
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QUESTiON  #  :  021 
DATABASE  #   5  037 

*  Accession  Number  Overlap  between  pairs  of  Searchers 
*********************************** 

Search  Evaiaation  : 


*** 

Proportions  *** 

A: 

#  Relevant 

Gl  : 

F  in  over i  ap  between  SI &S2  / 

F 

for 

SI 

B: 

#  Partial ly  Relevant 

G2: 

F  in  over i  ap  between  SI &S2  / 

F 

for 

S2 

C:' 

#  Not  Relevant 

Hi  : 

b  in  over i  ap  between  SI SS2  / 

D 

for 

SI 

D: 

Total  #  Evaluated 

H2: 

D  in  overlap  between  SI aS2  / 

D 

for 

S2 

E: 

#  Not  Evaluated 

11  : 

A+B  in  overlap  between  S1&S2 

/ 

A+B 

for 

F: 

Total   #  Retrieved 

12: 

A+B  in  overlap  between  S1&S2 

/ 

A+B 

for 

Jl  : 

E  in  over 1 ap  between  Si  &S2  / 

E 

for 

SI 

J2: 

E  in  over 1 ap  between  Si  &S2  / 

E 

for 

S2 

Total   for  Question  : 


Searcher 

A 

B 

e 

D 

E 

F 

002 

6 

6 

9 

0 

IS 

1 

0 

0 

i 

d 

i 

gi6 

0 

2 

28 

30 

0 

30 

021 

0 

1 

.  B 

6 

0 

6 

033 

11 

3 

31 

45 

0 

45 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

e 

D 

E 

F 

Gl 

Q2 

HI 

H2 

11 

12 

Jl 

002 

014 

0 

g 

d 

d 

d 

0 

0 

.000 

0 

.000 

0 

.000 

0.000 

0 

.000 

0 

.000 

0. 

ggg 

g.gi 

002 

016 

0 

g 

0 

d 

d 

0 

0 

.odo 

0 

.000 

0 

.  000 

0.000 

0.000 

0 

.000 

0. 

000 

g.gi 

002 

021 

0 

0 

g 

0 

d 

0 

g 

.ddd 

d 

.doo 

0 

.000 

0.000 

0 

.  000 

0 

.  000 

0. 

000 

0.01 

002 

033 

2 

g 

g 

2 

0 

2 

0 

.133 

d 

.044 

0 

.133 

0.044 

0 

.  333 

0 

.142 

0. 

000 

0.01 

014 

gi6 

g 

g 

0 

^  0 

0 

0 

0 

.000 

0 

.000 

0 

.000 

0.000 

0 

.000 

0 

.000 

0. 

000 

0.0( 

014 

021 

0 

0 

g 

g 

g 

g 

0 

.000 

0 

.000 

0 

.ogo 

0.000 

0 

.000 

0 

.000 

0. 

000 

0.0( 

014 

033 

0 

0 

0 

g 

g 

g 

g 

.ggg 

0 

.000 

0 

.000 

0.000 

0 

.000 

0 

.000 

0. 

000 

0.0( 

01 6 

021 

0 

0 

0 

0 

0 

0 

g 

.ggg 

g 

.ggo 

g 

.coo 

0.000 

0 

.000 

0 

.000 

0. 

000 

o.oc 

016 

033 

0 

0 

0 

0 

0 

0 

0 

.000 

0 

.000 

0 

.ggg 

0.000 

0 

.000 

0 

.000 

0. 

300 

g.dc 

021 

033 

0 

0 

0 

0 

0 

0 

0 

-000 

0 

.000 

0 

.000 

0.000 

0 

.000 

0 

.000 

0. 

000 

Q.oe 

445 

o 
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QUESTION  #  :  022 
DATABASE  #   :  108 


*   J^Gps^sion  Number^^P  pairs  of  Searchers  * 


Search  Evaiaation 


A: 

#  Relevant 

SI  : 

B: 

#  Partially  Relevant 

62: 

C: 

8  Not  Relevant 

Rl : 

D: 

To ta 1  #  Eva 1 ua ted 

82: 

E: 

#  Not  Evaluated 

11 : 

F: 

Total   #  Retrieved 

12: 

Jl: 

J2: 

***     Proportions  *** 

F  in  overlap  between  Sl&Si 


/  F  for  SI 


F  in  overlap  between  SI &S2  /  F  for  S2 

D  in  overlap  between  Si &S2  /  D  for  SI 

D  in  overlap  between  SI &S2  /  D  for  S2 

A+B  in  overlap  between  S1&S2  /  A+B  for  Si 

A+B  in  overlap  between  Si&S2  /  A+B  for  S2 

E  in  overlap  between  S1&S2  /  E  for  SI 

E  in  over 1 ap  between  SI &S2  /  E  f or  S2 


Total  for 

Question 

■ 

Searcher 

A 

B 

C 

D 

E 

'  F 

008 

2 

18 

g 

20 

S3 

1.03 

027 

6 

60 

g 

66 

68 

134 

032 

1 

5 

0 

6 

8 

14 

038 

1 

5 

0 

6 

11 

17 

039 

0 

6 

0 

6 

64 

70 

Searcher  Overlap  : 


Searcher 


SI 

ggs 

S2 
g27 

A 

D 

B 
3 

G 
0 

D 
3 

E 
2 

F 
5 

Gi 

62 

HI 

H2 

11 

12 

Jl 

0. 

0:2 

0. 

048 

0- 

037 

0 

-150 

0 

.045 

0 

.150 

0 

.045 

0 

-024 

008 

g32 

0 

g 

0 

0 

0 

0 

0- 

000 

0- 

000 

Q 

-000 

0 

-000 

0.000 

0 

.000 

0 

.000 

0. 

oc 

008 

038 

0 

0 

0 

g 

0 

0 

goo 

000 

0 

-000 

0 

.000 

0 

.000 

0 

.000 

0 

.000 

0. 

oe 

008 

g39 

6 

0 

0 

0 

1 

1 

g. 

009 

g. 

□14 

g 

-goo 

g 

.000 

d 

.000 

g 

.080 

0 

.012 

0. 

01 

027 

032 

1 

4 

0 

5 

5 

10 

0. 

074 

0. 

714 

0 

-075 

0 

.  833 

0 

.075 

0 

.  833 

g 

.073 

0. 

62 

027 

038 

1 

5 

0 

6 

3 

9 

0. 

067 

0. 

529 

g 

-g9g 

1 

.□□0 

D 

.g9g 

1 

-000 

0 

.044 

0. 

27 

027 

03? 

0 

4 

0 

4 

5 

.  9 

d. 

067 

0- 

128 

0 

-060 

0 

.  666 

□ 

.060 

g 

.666 

0 

-073 

0. 

07 

032 

038 

0 

1 

0 

1 

0 

1 

0- 

071 

0- 

058 

0 

.166 

0 

.166 

□ 

.166 

0 

.166 

g 

.gog 

g. 

oc 

032 

039 

0 

s 

0 

5 

8 

13 

0. 

928 

0- 

185 

0 

-  833 

0 

.  833 

0 

.  833 

n 

.  833 

1 

.000 

0. 

12 

038 

039 

0 

1 

0 

1 

0 

1 

Q- 

058 

0. 

014 

0 

-166 

0 

.166 

0 

.166 

0 

.166 

0 

.000 

0. 

00 

ERIC 


iSUEStiON  #  :  (323 
DATABASE  #   :  032 


*     Accession  Number  Overlap  between  pairs  of   Searchers  * 


Search  Eva 1 nation 


_  ***     Proportions  *** 

A:   #  Relevant     _  Gl :  F  in  overlap  between  Si&S2  /  F  for  SI 

B:   #  Partially  Relevant  G'2:  F  in  overlap  between  S1&S2  /  F  for  S2 

C:   f  Not  Relevant  HI :  D  in  overlap  between  Si&S2  /   D  for  SI 

p:   Total   #  Evaluated         H2:  D  in  overlap  between  S1^S2  /   D  for  S2 

E:   #  Not  Evaluated  II:  A+B  in  overlap  between  S1&S2  /  A-*-B  for  Si 

F:   Total  #  Retrieved         12:  A+B  in  overlap  between  S1&S2  /  A-*-B  for  S2 

Jl  :  E  in  overlap  between  Si&S2  /  E  for  Si 

J2:  E  in  overlap  between  S1&S2  /  E  for  S2 


Total   for  Question  : 


Searcher       A       B       C       D       E  F 


□15 

23 

3 

9 

35 

0 

35 

030 

1 

1 

4 

_6 

D 

6 

034 

16 

2 

6 

24 

0 

24 

035 

3 

4 

28 

35 

0 

35 

041 

13 

0 

6 

21 

0 

21 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

e 

D 

E 

F 

61 

62 

HI 

H2 

11 

12 

Jl 

j: 

015 

030 

1 

0 

p 

1 

0 

1 

0 

-028 

0. 

166 

0- 

628 

0. 

166 

0. 

038 

0 

.5G0 

0 

015 

034 

16 

1 

5 

22 

P 

22 

0 

.628 

P. 

916 

6. 

628 

b. 

916 

0. 

653 

0 

.  944 

0 

.  C  CJ 

015 

035 

2 

0 

1 

3 

P 

3 

P 

-085 

□  . 

085 

0- 

685 

0. 

085 

076 

0 

.285 

tj 

f  i^or 

Q 

.OCi 

015 

041 

6 

0 

7 

15 

0 

15 

0 

.428 

P. 

714 

0. 

428 

0. 

714 

d. 

307 

P 

.6i5 

C 

.  '  /'-id 

0 

-OOf 

030 

034 

1 

0 

0 

1 

0 

1 

0 

.166 

p. 

P41 

p. 

166 

P. 

041 

0. 

5PP 

G 

.055 

0 

.obc 

030 

035 

1 

0 

3 

4 

0 

4 

0 

.666 

0. 

114 

p. 

666 

p. 

114 

0. 

500 

0 

.142 

G 

.C3Q 

C 

030 

041 

i 

b 

b 

1 

0 

1 

0 

.166 

0. 

047 

0. 

166 

p. 

P47 

P. 

SOO 

□ 

.076 

0 

.  DC"» 

.adc 

034 

035 

2 

b 

b 

2 

0 

2 

0 

.083 

0. 

057 

0. 

083 

0. 

057 

0. 

111 

0 

.2SE 

0 

.□DO 

0 

.HOC 

034 

041 

G 

4 

8 

0 

8 

0 

•  333 

0. 

380 

b. 

333 

b. 

380 

0. 

222 

0 

.3D7 

0 

.ODQ 

.  ODC 

035 

041 

1 

0 

1 

2 

0 

2 

0 

.057 

0. 

095 

0. 

057 

0. 

095 

0. 

142 

0 

.076 

0 

.000 

0. 

DOC 
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QUESTION  #  :  024 
DATABASE  #   :  191 


*     Accession  Number  Overlap  between  pairs  of   Searchers  * 


Search  Evaluation 


Proportions  *** 

A: 

#  Relevant 

Gl  : 

F  in  overlap  between  Si&S2  / 

F 

for 

SI 

B: 

#  Partial ly  Relevant 

G2: 

F  in  overlap  between  Si&S2  / 

F 

for 

S2 

C: 

#  Not  Relevant 

HI  : 

P  in  overlap  between  Si &S2  / 

B 

for 

SI 

D: 

Total   #  Evaluated 

H2: 

P  in  overlap  between  S1&S2  / 

for 

S2 

£: 

#  Not  Evaluated 

11 : 

A+B  in  overlap  between  SI &S2 

/ 

A+B 

f  CDT 

F: 

Total   #  Retrieved 

12: 

A+B  in  overlap  between  Si &S2 

/ 

A+B 

for 

Jl  : 

E  in  overlap  between  S1SS2  / 

E 

f  cr 

SI 

J2: 

E  in  overlap  between  31&S2  / 

E 

for 

S2 

Total   for  Question  : 


Searcher 

A 

B 

C 

D 

E 

F 

□07 

1 

2 

7 

1^ 

□ 

1^ 

□12 

1 

1 

6 

□ 

h 

□23 

□ 

1 

1^ 

11 

□ 

11 

□25 

□ 

□ 

□ 

□ 

1 

1 

□36 

1 

2 

S 

11 

□ 

11 

Searcher  Overlap 


Searcher 


SI 

S2 

A 

B 

e 

D 

E 

F 

Gi 

G2 

HI 

H2 

il 

12 

Jl 

□07 

□12 

1 

1 

1 

3 

□ 

3 

g. 

3og 

g 

-sgg 

g 

-  sgg 

g. 

.66& 

1 

.□□□ 

□ 

.□□□ 

□07 

□23 

g 

D 

□ 

□ 

□ 

□ 

□  - 

□□□ 

□ 

-□□6 

0 

.GOO 

□  - 

□ 

.□□0 

0 

.□□□ 

0 

.□00 

□07 

□25 

□ 

□ 

□ 

□ 

□ 

g 

g. 

ggg 

□ 

.gog 

g 

.COD 

g. 

QOO 

U 

.□gg 

d 

.□og 

6 

.ggg 

□□7 

□36 

1 

1 

5 

7 

□ 

7 

g. 

7gg 

g 

-  636 

D 

.7CC 

g. 

0 

.666 

0 

.666 

g 

.ggg 

□12 

□23 

□ 

□ 

□ 

□ 

g 

g 

□  . 

(:□□ 

g 

.ggg 

□ 

.gcc 

n 

wOog 

0 

.ggo 

c 

.000 

□12 

□25 

□ 

□ 

□ 

□ 

□ 

□ 

c. 

□□□ 

□ 

.g^g 

g 

.  □DC 

0. 

goo 

G 

.ggg 

g 

.□og 

g 

.ggg 

□12 

□3d 

1 

1 

2 

□ 

□  . 

666 

□ 

.  363 

□ 

.666 

363 

'\ 

.□□□ 

□ 

•  6o6 

g 

.ggg 

□23 

025 

□ 

□ 

□ 

□ 

□ 

□ 

□  - 

□□□ 

□ 

.□□□ 

□ 

.  □□□ 

0. 

□□G 

0 

.□□0 

0 

.□00 

□ 

.000 

□23 

036 

0 

□ 

□ 

□ 

□ 

□ 

□  - 

□□□ 

□ 

.□□□ 

□ 

-□□□ 

0. 

□□O 

.  coo 

0 

.  Of>0 

D 

.000 

□25 

□36 

0 

□ 

0 

□ 

□ 

□ 

□  . 

□□□ 

□ 

.□□□ 

□ 

.□□□ 

0, 

□ 

r?oo 

0 

0 

.000 

o.e 


ERIC 


QUESTION  #  :  025 
DATABASE  #   :  001 


******************************************** 

*  Accession  Namber  Overlap  between  pairs  of  Searchers  * 
****************************************** 


Search  Eval uatibh  : 


As   #  Relevant 


***  Proportions  *** 

Gl  :  f="  in  overlap  between  S1SS2  /  F  f  or  SI 

B:   #  Partial ly  Relevant  G2:  F  in  overlap  between  S1SS2  /  F  for  S2 

HI?  D  in  overlap  between  SldS2  /   D  for  SI 

H2:  D  in  overlap  between  SiaS2  /  D  for  S2 


0:  #  Not  Relevant 

b:  Total  #  Evaluated 

E:  #  Not  Evaluated 

Fs  Total   #  Retrieved 


J;l:  A+B  in  overlap  between  S1&S2  /  A+B  for  SI 

12:  A+B  in  overlap  between  SI 6S2  /  A+B  for  S2 

Ji :  E  in  overlap  between  S1&S2  /  E  for  SI 

J2?  E  in  overlap  between  S1&S2  /  E  for  92 


Total   for  Question  : 


Searcher       A  B 


□02 

16 

15 

014 

3 

2 

016 

0 

0 

021 

1 

0 

033 

0 

0 

C 

D 

E 

F 

19 

50 

199 

249 

6 

11 

48 

59 

3 

3 

33 

36 

0 

1 

6 

7 

0 

0 

i 

i 

Searcher  Overlap 
Searcher 


SI 

S2 

A 

B 

c 

D 

E 

F 

Gl 

G2 

HI 

H2 

11 

12 

Jl 

002 

014 

1 

0 

0 

1 

14 

15 

0 

.060 

0. 

254 

0. 

020 

b. 

b9b 

b.b32 

0 

.200 

0.070 

0.2< 

002 

016 

0 

0 

0 

0 

0 

b 

0 

.000 

b. 

000 

0. 

000 

0. 

000 

0.000 

0 

.oob 

0.000 

0,0( 

002 

021 

b 

b 

0 

0 

1 

1 

0 

.004 

0. 

142 

0. 

000 

0. 

000 

0.000 

0 

.000 

0 . 005 

002 

033 

b 

0 

0 

0 

0 

0 

0 

.000 

0. 

000 

0. 

000 

0. 

000 

0.000 

0 

.000 

0.000 

o.oc 

014 

016 

d 

0 

d 

d 

0 

0 

0.000 

0. 

000 

0. 

000 

0. 

000 

0.000 

0 

.000 

0.000 

o.oc 

014 

021 

d 

d 

d 

0 

i 

1 

0 

.016 

0. 

142 

0. 

000 

0. 

000 

O.OOG 

0 

.GOO 

0.020. 

G.li! 

014 

033 

d 

0 

d 

d 

d 

0 

0 

.000 

0. 

000 

0. 

GOO 

0. 

GOO 

G.OGG 

0 

.OGO 

0,000 

o.oe 

016 

□21 

D 

0 

d 

d 

d 

0 

0 

.000 

0*. 

000 

0. 

000 

0. 

000 

O.OGG 

G 

iOOO 

0.000 

G.oe 

016 

033 

□ 

u 

d 

0 

0 

0 

0 

-doo 

0. 

000 

0. 

000 

0. 

000 

0.000 

0 

.000 

o.ooo 

G.OG 

021 

033 

0 

0 

0 

0 

0 

0 

0 

.000 

0. 

000 

0. 

000 

0. 

000 

0.000 

0 

.000 

0.000 

o.oc 
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QUESTION  #  :  026 
DATABASE  #   :  038 


*     Accession  Number  Overlap  between  pairs  of  Searchers  * 
Search  Evaluation  : 


*** 

Prbpbrtiphs  *** 

A: 

#  Relevant 

Gl  : 

F  in  overlap  between  SI aS2  / 

F 

for 

SI 

B: 

#  Partially  Relevant 

G2: 

F  in  overlap  between  SI SS2  / 

F 

for 

S2 

G: 

#  Not  Relevant 

HI  : 

D  in  overlap  between  S15S2  / 

D 

for 

81 

D: 

Total  #  Evaluated 

H2: 

b  in  overlap  between  S1SS2  / 

D 

for 

S2 

E: 

#  Not  Evaluated 

il  : 

A+B  in  overlap  between  S1SS2 

/ 

A+B 

for 

F: 

Total  #  Retrieved 

12: 

A+B  in  overlap  between  S1SS2 

/ 

A+B 

for 

Jl  : 

E  in  overlap  between  S1&S2  / 

E 

for 

SI 

J2: 

E  in  overlap  between  S1&S2  / 

E 

1  or 

S2 

nt:kl   for  Question  : 


3*:?archer 

A 

B 

C 

D 

E 

F 

1 

1 

0 

1 

015 

0 

a 

3 

3 

0 

3 

023 

8 

0 

27 

d 

27 

025 

9 

2 

d 

ii 

d 

11 

036 

1 

1 

0 

2 

0 

2 

Searcher 


SI 

S2 

A 

B 

C 

D 

E 

F 

61 

012 

di5 

0 

0 

0 

0 

0 

0 

0-000 

012 

023 

0 

d 

0 

0 

Q 

0 

0 . 000 

012 

025 

1 

d 

d 

1 

0 

1 

1  .000 

□12 

036 

1 

0 

d 

1 

0 

1 

i .  000 

□15 

023 

0 

0 

d 

d 

d 

d 

d.ddd 

015 

d25 

D 

0 

0 

0 

d 

d 

d.ddd 

015 

036 

0 

D 

0 

0 

0 

0 

d.doo 

023 

025 

1 

□ 

0 

1 

d 

1 

0,037 

023 

036 

0 

0 

0 

0 

□ 

0 

o.ooo 

025 

036 

1 

0 

0 

1 

0 

1 

0,090 

G2         HI         H2         11         12  Ji 

0.000  0,000  d-ddd  d-Sdd  □,dDD  □.□□d  d.oi 

0,000  0.000  0.000  d-000  □,ddd  d.Pdd  □.□( 

0.090  1.000  0.090  1,000  0,090  0,dOO  0,Q\ 

0.500  1.000  0-500  l.OOd  0.500  0.000  □.□( 

d.ddo  Q.ooo  0.000  o.ono  o.ooo  o.ooo  o.o( 

d.ddd  o.ooo  o.ooo  o.ooo  o.ooo  o.ooo  o.dt 

d.ddd  d.doo  d.ddd  o.oeo  o.ooo  o.ooo  o.dc 

d.d9d  d.d37  d.d9d  0.037  0.090  0.000  o.oc 

0,000  d,ddd  d.ddd  d.ddd  d.odo  d.ooj  0.0c 

0.500  0.090  0.500  0.090  0.500  0.000  C.OC 
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QUESTION  #  :  027 
DATABASE  #    S  CD13 


*  Accession  Number  Overlap  between  pairs  of  Searchers  * 
************************ 


Search  Evaluation  : 


*** 

A: 

#  Relevant 

Gl : 

B: 

#  Partial ly  Relevant 

G2: 

C: 

#  Not  Relevant 

His 

D: 

Total  #  Evaluated 

H2: 

E: 

#  Not  Evaluated 

11: 

f=": 

Total  #  Retrieved 

12: 

Jl  : 

J2: 

***     Prdpbrtiphs     ***    _  _ 

F  in  overlap  between  S15S2  /  F  for  SI 
F  in  overlap  .between  S1SS2  /  F  for  S2 
D  in  overlap  between  SI 5S2  /  D  for  SI 
D  in  overlap  between  SiaS2  /  D  for  S2 
A+B  in  overlap  between  S1SS2  /  A+B  for 
A+B  in  overlap  between  SI aS2  /  A+B  for 
E  in  overlap  between  S1&S2  /  E  for  SI 
E  in  overlap  between  Si&S2  /  E  for  S2 


SI 
S2 


Total   for  Question 


Searcher 

A 

B 

C 

D 

E 

F 

□05 

7 

11 

1 

19 

12 

31 

□11 

7 

19 

13 

39 

15 

54 

□25 

22 

39 

32 

93 

34 

127 

□27 

8 

12 

11 

31 

7 

38 

□4^ 

19 

14 

2 

35 

12 

47 

Searcher  Overlap 


Searcher 


SI 

S2 

A 

B 

C 

D 

E 

F 

Gl 

G2 

H-1 

H2 

11 

12 

Jl. 

□□5 

□11 

□ 

□ 

□ 

□ 

□ 

□ 

□  . 

□□□ 

□ .  □□CD 

O.OOu 

0. 

000 

0 

.000 

0 

.000 

0. 

000 

□□5 

□25 

1 

□ 

□ 

1 

2 

3 

□  - 

□96 

□  .□23 

0.05i 

0. 

010 

0 

.055 

0 

.016 

0. 

166 

DOS 

□27 

□ 

□ 

□ 

0 

2 

2 

□  . 

□64 

□  .□52 

0.000 

0. 

000 

0 

.000 

0 

.000 

0. 

166 

□□5 

□4^ 

1 

□ 

□ 

i 

2 

3 

0. 

□96 

□  .063 

0.052 

0. 

028 

0 

.055 

0 

.030 

0. 

166 

□11 

□25 

□ 

G 

G 

i 

1 

□  . 

□IS 

□  .□Q7 

0.000 

0. 

000 

0 

.000 

0 

.000 

0. 

066 

Oil 

□27 

□ 

□ 

□ 

□ 

S 

0. 

□□□ 

□  .GOO 

0.000 

0. 

000 

0 

.000 

0 

.000 

0. 

000 

Oil 

□40 

□ 

□ 

□ 

□ 

1 

1 

□  . 

□18 

□  .021 

d.ooo 

0. 

doo 

d 

.000 

0 

.ddd 

d. 

066 

□25 

□27 

8 

12 

11 

31 

7 

38 

□  . 

299 

1  .  doo 

d  •  333 

1 . 

ddo 

d 

.327 

i 

.dod 

d. 

205 

□25 

□4^ 

19 

14 

2 

35 

12 

47 

370 

1  . 000 

d.376 

1. 

odo 

d 

.54d 

1 

.odo 

d. 

352 

□27 

□4^ 

7 

5 

□ 

12 

2 

14 

□  . 

368 

0.297 

0.387 

0. 

342 

0 

.600 

0 

» 363 

0. 

285 

0.0€ 


1  .00 
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QUESTION  #  :  02S 
DATABASE  #   :  038 


*     Accession  Number  Overlap  between  pairs  of  Searchers  * 


Search  Evaluation 


A:   #  Relevant 

B:   #  Partially  Relevant 

C:   #  Not  Relevant  

E>:  Total  #  Evaluated 
E:   #  Not  Evaluated 
F:   Total  #  Retrieved 


***  _Propbrtiqns  *** 
Gl :  F  in  overlap  between  S1&S2  /  F 
G2:  F  in  overlap  between  S1&S2  /  F 
HI:  p  in  overlap  between  S15S2  /  p 
P  overlap  between  S1SS2_/  D 
ii :  A+B  in  overlap  between  S1SS2  / 
i2:  A+B  in  overlap  between  SI SS2  / 
Jl :  E  in  overlap  between  S1&S2  /  E  for  SI 
J2:   E  in  overlap  between  S1&S2  /  E  for  S2 


for  Si 
for'  S2 
for  SI 
for  S2 
A+B  for  SI 
A+B  for  S2 


Total   for  Question  : 


Searcher 

A 

B 

c 

D 

£ 

F 

007 

4 

14 

IS 

36 

d 

36 

Q12 

4 

S 

11 

0 

11 

023 

Q 

1 

2 

3 

0 

3 

034 

i 

1 

12 

14 

0 

14 

036 

0 

0 

0 

0 

1 

1 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

c 

D 

E 

F 

Gl 

G2 

HI 

H2 

ii 

12 

Ji 

J 

0Q7 

012 

4 

5 

2 

11 

0 

11 

0.305 

1 

.000 

0 

-305 

1 

.000 

0. 

500 

i 

.dOd 

g 

.odd 

d 

.bb 

007 

023 

0 

1 

1 

2 

0 

2 

0.055 

0 

.666 

0 

.055 

0 

.666 

0. 

055 

.1 

.gog 

g 

.oog 

d 

.00 

Q07 

034 

0 

0 

0 

0 

0 

0 

o.obo 

0 

.000 

0 

-000 

g 

.goo 

g. 

ggo 

g 

.000 

0 

.000 

g 

.00 

6Q7 

036 

0 

0 

0 

0 

0 

0 

0 . 000 

0 

.00C3 

0 

.000 

0 

.000 

0. 

000 

g 

-ggg 

g 

.ggg 

g 

.go 

012 

023 

d 

d 

0 

0 

b 

0 

0.000 

0 

-000 

0 

.000 

0 

.000 

0. 

ggg 

g 

.ggg 

g 

.ggg 

g 

.ggi 

012 

034 

d 

0 

0 

d 

0 

0 

0.000 

0 

.000 

0 

.000 

0 

.000 

0. 

000 

0 

.000 

0 

.000 

0 

-OOi 

012 

036 

0 

0 

d 

d 

b 

b 

0.000 

0 

.000 

0 

.000 

0 

.000 

0. 

000 

0 

.000 

0 

.000 

0 

-OOi 

023 

□34 

0 

0 

d 

d 

b 

d 

o.ood 

0 

.000 

0 

.000 

o.ooo 

0. 

000 

0 

.000 

0 

.000 

0 

.001 

023 

036 

0 

0 

0 

d 

d 

0 

0.000 

0 

.000 

0 

.000 

0 

.000 

0. 

000 

0 

.000 

0 

.000 

0 

.001 

034 

036 

0 

0 

0 

0 

0 

0 

O.GOO 

0 

-000 

0 

.000 

0 

-000 

0. 

000 

0 

.000 

0 

.000 

0 

.  OOI 

ERIC 
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QOEStidN  #  :  u29 
DATABASE  #   :  QOB 

*     Accession  Number  Overlap  between  pairs  of   Searchers  * 
Search  Evaluation  : 


A:   #  Relevant 

B:  #  Partially  Relevant 

C:   #  Not  Relevant 

D:   Total  #  Evaluated 

Es  #  Not  Evaluated 

F:   Total  #  Retrieved 


Gi 
G2 
Hi 
H2 
11 
12 
Jl 
J2 


Proportions 
F  in  overlap 
F  in  overlap 
D  in  overlap 

A+B  in  overlap 
A+B  in  overlap 


*** 
between 
between 
between 
between 
between 
between 


SI  5S2 
SI  SS2 
S1SS2 
S1SS2__ 
SI  SS2 
SI  SS2 


E 
E 


in 
in 


overlap 
overlap 


between 
between 


S1SS2 
SI  &S2 


F 
F 
D 
D 
/ 
/ 
E 
E 


for 
for 
for 
for 
fi+B 
A+B 
for 
for 


Si 

S2 

SI 

92 

for 

for 

SI 

S2 


SI 
S2 


Total   for  Question  : 


Searcher       A       B       6       D       E  F 


Oil 

8 

8 

12 

28 

22 

50 

027 

15 

13 

50 

78 

23 

101 

038 

3 

1 

1 

5 

3 

8 

039 

5 

3 

2 

10 

5 

15 

040 

3 

0 

0 

3 

0 

3 

Searcher  Overleap 
Searcher 


SI 

S2 

A 

B 

e 

D 

E 

oil 

027 

5 

3 

1 

9 

5 

oil 

038 

3 

1 

1 

5 

3 

oil 

039 

1 

2 

1 

4 

4 

oil 

040 

0 

0 

2 

0 

027 

038 

3 

1 

1 

5 

3 

027 

039 

2 

2 

1 

5 

3 

027 

040 

3 

0 

0 

3 

0 

□38 

039 

1 

d 

i 

2 

1 

03? 

□40 

1 

d 

d 

1 

0 

039 

040  • 

1 

0 

0 

1 

0 

F 

Gl 

G2 

HI 

H2 

11 

12 

ji 

J 

14 

0- 

280 

0 

.138 

0. 

321 

0 

.115 

g. 

500 

0. 

285 

g. 

227 

0 

.21 

8 

0. 

16D 

1 

.000 

0. 

178 

1 

.000 

0. 

250 

1  . 

000 

g. 

136 

1 

.dd 

8 

0. 

160 

0 

.533 

0. 

142 

0 

.400 

0. 

137 

0. 

375 

g. 

181 

0 

.80 

2 

0. 

040 

0 

.666 

0. 

071 

0 

.666 

0. 

125 

0. 

666 

g. 

ogg 

0 

.ggi 

a 

g. 

g79 

1 

.Odd 

0. 

064 

i 

.000 

0. 

142 

1  . 

000 

0. 

130 

1 

.ogi 

8 

g. 

079 

0 

.  533 

g. 

064 

0 

.500 

0. 

142 

0. 

500 

0. 

130 

0 

.601 

0. 

029 

1 

.goo 

0. 

038 

1 

.ddd 

0. 

107 

i . 

000 

0. 

000 

0 

.001 

3 

0. 

375 

0 

.2go 

g. 

4gg 

0 

.200 

0. 

250 

0. 

125 

0. 

333 

0 

.20( 

1 

0. 

125 

0 

-  333 

g. 

2gg 

g 

.  333 

0. 

250 

0. 

333 

0. 

000 

0 

.ooc 

1 

0. 

066 

0 

.  333 

0. 

100 

0 

.  333 

0. 

125 

0. 

333 

0. 

000 

0 

.ooc 

ERIC 


453 


QUESTION  #  :  030 
DATABASE  #   :  071 

*  f ipp  Nuniber  Overlap  between  pairs  of  Searchers  * 

Search  Evaiaation  : 


Proportions  *** 

A: 

#  Relevant 

Gi  : 

F  in  overlap  between  SlfiS2  / 

F 

for 

SI 

B: 

#  Partially  Relevant 

€2: 

F  in  overlap  between  S1SS2  / 

F 

for 

S2 

C: 

g  Not  Relevant 

HI  : 

D  in  overlap  between  Si&S2  / 

D 

for 

SI 

D: 

Total   1  Evaluated 

H2: 

0  in  overlap  between  S1&S2  / 

D 

for 

S2 

E: 

»  Not  Evaluated 

11  : 

A+B  in  overlap  between  S1&S2 

/ 

A+B 

for 

F: 

Total  #  Retrieved 

12: 

A+B  in  overlap  between  S1^S2 

/ 

A+B 

for 

Jl  : 

E  in  overlap  between  SI aS2  / 

E 

for 

SI 

J2: 

E  i.n  overlap  between  SiaS2  / 

E 

far 

S2 

Total   for  Question  : 


Searcher 

A 

B 

e 

D 

E 

F 

007 

7 

1 

2 

10 

d 

10 

gi2 

29 

14 

5 

48 

d 

48 

023 

37 

15 

5 

56 

d 

56 

025 

2 

0 

g 

2 

d 

2 

036 

1 

0 

0 

1 

0 

1 

Searcher  Overlap 


Searcher 


SI 

S2 

A 

B 

e 

D 

E 

F 

61 

007 

012 

1 

d 

p 

1 

d 

1 

0-100 

007 

023 

1 

0 

d 

1 

d 

1 

d-iod 

007 

025 

0 

0 

0 

d 

d 

d 

d-000 

007 

036 

0 

0 

g 

0 

0 

0 

0.000 

012 

023 

29 

12 

3 

44 

g 

44 

g-916 

012 

025 

1 

0 

0 

1 

0 

1 

g.g2g 

012 

036 

0 

0 

0 

0 

0 

0 

0,000 

023 

025 

2 

0 

0 

2 

0 

2 

0.035 

d23 

036 

1 

d 

0 

i 

0 

1 

0.017 

025 

036 

1 

0 

0 

1 

0 

i 

0.500 

0.020 

d .  di  7 
a .  ddd 

D.OOd 
d-7S5 
d-Sdd 

o.doo 

1  . 000 
1  . 000 
1  .000 


HI 

b.loo 
d.ido 
d.dod 
d.dod 

0.916 

o.d2d 
d-ddd 

0.g35 
0.D17 
0.500 


H2 

0.020 
0.017 

d.ddd 
d.odd 

d .  785 

Sdd 

000 


d 
o 

1  . 000 


goo 

000 


II 

0.125 
0.125 

d  -  ddo 
d.doo 

d.953 
d .  023 

d-doo 

d-d39 
d-019 
0.500 


0.023 
0.019 
0 . 000 
0.000 
d  .803 

d.sdd 

0.  ddd 

1  .ODD 

1 .  ddd 

1  . 000 


Jl 

0-000  o.c 
0.000  d.c 

0.000  0;C 

o.doo  o.c 
d.Qdo  o.e 
d.ddb  o.B 
d.ddd  o.d 
d-ddd  o.d 
d-ddo  d.D 

0.000  0.0 


QUESTION  #  :  031 
DATABASE  #   :  061 


*     Accession  Number  Overlap  between  pairs  of  Searchers  * 


Search  Evaluation  : 


A:  #  Relevant 

B:  #  Partially  Relevant 

C:  g  Not  Relevant 

D:  Tot^al  g  Evaluated 

E:  #  Not  Evaluated 

F:  Total  #  Retrieved 


***  iProportions 

Gl :  F  in  overlap  between  Si&S2  / 

G2:  F  in  overleap  between  Si&S2  ^ 

HI:  D  in  overlap  between  S1&S2  / 

H2:  P  in  overlap  between  SI &S2  / 

II:  A+B  in  overlap  between  S1SS2 

12:  A+B  in  overlap  between  SI 5S2 

Jl :  E  in  overlap  between  SI SS2  /  E  f or  SI 

J2:  E  in  overlap  between  S1&S2  /  E  for  S2 


F 

,f  or 

Si 

F 

for 

S2 

t) 

for 

Si 

D 

for 

S2 

/ 

A+B 

for 

SI 

/ 

A+B 

for 

S2 

E 

for 

SI 

E 

for 

S2 

Total   for  Question 


Searcher       A  B 


go7 

1 

a 

gi2 

6 

6 

023 

10 

12 

025 

5 

4 

036 

3 

0 

e 

D 

E 

F 

19 

20 

0 

20 

34 

46 

0 

46 

41 

63 

g 

63 

IS 

27 

0 

27 

8 

11 

0 

11 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

e 

D 

E 

Gi 

G2 

Hi 

H2 

11 

12 

Jl 

gg7 

gi2 

g 

0 

0 

0 

0 

0 

0. 

000 

0. 

000 

0. 

000 

0 

.000 

0. 

GOO 

0 

.000 

0. 

000 

0 

.oe 

goz 

023 

g 

g 

0 

g 

g 

g 

g.aga 

ggg 

g. 

ggg 

g 

.ggg 

g. 

ggg 

g 

.ggg 

g. 

ggg 

d 

.d( 

007 

025 

g 

g 

g 

g 

g 

g 

g. 

ggg 

g. 

ggg 

g. 

ggg 

0 

.ago 

g. 

gog 

0 

.ggg 

g. 

ggg 

g 

.gc 

007 

036 

0 

0 

0 

0 

0 

0 

0. 

000 

0. 

000 

0. 

ODO 

0 

.000 

0. 

000 

0 

.000 

0. 

000 

g 

.gc 

012 

023 

5 

4 

13 

22 

0 

22 

0. 

478 

g; 

349 

0. 

478 

0 

.349 

0. 

750 

0 

.409 

0. 

000 

0 

.oc 

012 

025 

5 

4 

13 

22 

0 

22 

0- 

478 

0. 

814 

0. 

478 

0 

.814 

0. 

750 

1 

.000 

0. 

000 

0 

.oc 

012 

036 

3 

0 

a 

11 

0 

11 

0- 

239 

1  - 

000 

0. 

239 

1 

.000 

0. 

250 

1 

.000 

0. 

000 

0 

.oc 

023 

025 

5 

4 

18 

27 

0 

27 

0- 

428 

1  - 

000 

0. 

428 

1 

-OCDO 

0. 

4CD9 

1 

.000 

0. 

000 

0 

.oc 

023 

036 

2 

0 

7 

9 

0 

9 

0. 

142 

0- 

818 

0- 

142 

0 

.818 

0- 

090 

0 

.666 

0- 

000 

0 

.oc 

025 

036 

2 

0 

7 

9 

0 

9 

0. 

333 

0. 

818 

0. 

333 

0.818 

0. 

222 

0 

.666 

0. 

000 

0 

.oc 

455 

ERIC 


QUESTiOlvi  #  :  032 
DATABASE  #   :  DOS 


♦Accession  Number  Overlap  between  pairs  of  Searchers  * 


Search  Evaiaation  : 


***     Proportions  *** 
A:   #  Relevant  <Si  :   F  in  overlap  between  S15S2  /  F  for  SI 

B:  #  Partially  Relevant  02:  F  in  overlap  between  S15S2  /  F  for  S2 
C:   #  Not  Relevant  HI :   D  in  overlap  between  S15S2  /   D  for  SI 

D:   Total  #  Evaluated         H2:   D  in  overlap  between  S15S2_/  D  for  S2 
E:   #  Not  Evaluated  ii :   f^^-^B  in  overlap  between  S15S2  /  A+B  for  SI 

F:   Total   #  Retrieved         12:   A^«^   in  over  lap  between  SI  SS2  /  A+B  for  S2 

Jl  5   E   \in  overlap  between  SiaS2  /  E  f  or  SI 
^  in  overlap  between  S1&S2  /  E  for  S2 


Total   for  Question  : 


Searcher       A  B 


DDI 

6 

G 

□08 

42 

3 

□  17 

_0 

1 

02? 

34 

4 

032 

58 

2 

e 

D 

E 

F 

d 

6 

C 

6 

6 

45 

74 

119 

17 

IS 

0 

IS 

D 

38 

i  12 

150 

0 

60 

•119 

179 

Searcher  Overlap  : 


Searcher 


Si 

S2 

A 

B 

c 

D 

E 

F 

DDI 

D08 

4 

Q 

d 

4 

0 

4 

□01 

017 

0 

0 

d 

Q 

Q 

0 

DDI 

□29 

2 

0 

D 

2 

d 

2 

001 

032 

4 

D 

0 

4 

d 

4 

008 

017 

D 

D 

D 

0 

0 

G 

008 

029 

22 

2 

□ 

24 

A4 

68 

008 

032 

19 

0 

0 

▲ 

35 

54 

017 

029 

0 

6 

0 

0 

□ 

D 

017 

032 

0 

0 

0 

0 

0 

0 

£329 

032 

19 

0 

0 

19 

58 

77 

Gl          G2         HI          H2         11  12  Jl 

0.666  0.033  0.666  O . OSS  0.666  0.088  0.000  0.0! 

0.000  0.000  0.000  o.pgg  o-Qoo  o.ODO  o.ooo  o.Oi 

0.333  0.013  0.333  0.052  0.333  0.052  D-OOD  O.Oi 


0.666 
0.000 
0.533 


0.666  0.022  0.666  0.066 
•G.GOG  G.OOO  0.000  O.OOD 
0.57^i  d.453  G.533  0.631 
0.453  0.301   d.422  0.316  0.422 

0.000  0.000  o.ddd  Q.ddQ  d.ooo 

0-000  0.000  0.000  Q.OOG  O.GGd  0.000  0.000  O.OC 
0.513  0.430  0.500  0.316  0.500  0.316  0.517  0.4^ 


0.066  O-OOQ  O.Oi 

0.000  0.000  □.01 

0.631   0.594  g.3< 

0.316  0.472  0.2< 

0.000  0.000  O.OC 


ERIC 


QUESTiON  #  :  033 
DATABASE  #   :  008 

*     Accession  Number  Over! ap  between  pairs  of  Searchers  * 


Search  Evaluation 


A:  #  Relevant 

B:  #  F>artialiy  Relevant 

C:  #  Not  Relevant 

D:  Total  #  Evaluated 

E:  #  Not  Evaluated 

F:  Total  #  Retrieved 


Proportions 

Gl  : 

F  in  overlap  between  S1SS2  / 

F 

for 

SI 

G2s 

F  in  overlap  between  S15S2  / 

F 

for 

S2 

HI : 

D  in  overlap  between  S15S2_/ 

D 

for 

SI 

H2: 

D  in       over 1 ap  between  SI SS2 

/ 

D  for  S2 

ii  : 

A+B  iri  overlap  between  SI  SS2 

/ 

A+B 

for  SI 

12: 

A+B  in  overlap  between  SI SS2 

/ 

A+B 

for  S2 

Jl: 

E.in  overlap  between  Si&S2  / 

E 

for 

SI 

J2: 

E  in  overlap  between  Si&S2  / 

E 

for 

S2 

TP"^?!   for  Qu0stidh  : 


Searcher       A  B 


□ 

□ 

dos 

□ 

□ 

□17 

3 

□29 

1 

□ 

□32 

IB 

26 

C 

D 

E 

F 

□ 

□ 

1 

1 

3 

3 

2 

5 

Q 

3 

4 

7 

0 

i 

1 

6 

47 

38 

85 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

c 

D 

E 

F 

Gl 

G2 

HI 

H2 

11 

12 

Ji 

□□i 

□□8 

□ 

□ 

G 

□ 

□ 

□ 

.OGO 

0. 

GGG 

0 

.000 

g 

.ggg 

0. 

ggg 

0. 

ddd 

0. 

000 

0. 

0 

□01 

Q17 

□ 

□ 

□ 

n 

□ 

□ 

□ 

.GGG 

0. 

000 

0 

-000 

0 

.000 

g. 

ggg 

g. 

Odd 

0. 

Odd 

d. 

d 

□01 

D29 

d 

□ 

□ 

5 

□ 

G 

0 

.GGG 

0. 

000 

0 

.000 

0 

.000 

6. 

000 

0. 

000 

g. 

ggg 

0. 

g 

DDI 

D32 

D 

d 

□ 

□ 

□ 

□ 

□ 

.GOG 

0. 

000 

0 

.000 

0 

.  000 

0. 

000 

0. 

000 

0. 

000 

0. 

Oi 

□DS 

□17 

D 

□ 

□ 

d 

d 

□ 

□ 

.GGG 

0. 

000 

0 

.000 

0 

.000 

0. 

000 

0. 

000 

0. 

OCDO 

0. 

01 

goB 

□29 

0 

□ 

d 

d 

d 

□ 

G 

.GOD 

0. 

000 

0 

.000 

0 

.000 

0. 

000 

0. 

000 

0. 

000 

0. 

C3I 

□□8 

□32 

□ 

d 

□ 

□ 

d 

□ 

G 

.GGG 

G. 

GGG 

0 

.000 

0 

.000 

0- 

000 

0- 

000 

0. 

000 

0. 

01 

□17 

029 

□ 

g 

g 

□ 

□ 

□ 

□ 

.GOG 

G. 

GGG 

G 

.GGG 

0 

.000 

0. 

000 

0. 

000 

0. 

000 

0. 

01 

□17 

032 

Q 

□ 

□ 

□ 

□ 

□ 

□ 

.dOO 

d. 

doo 

0 

.dGd 

0 

.doo 

0. 

□do 

0. 

doo 

0. 

000 

0- 

0( 

□29 

□32 

1 

□ 

□ 

1 

□ 

1 

G 

.500 

0. 

Oil 

1 

.000 

0 

.021 

1 . 

000 

0. 

024 

0. 

000 

Q. 

0( 

457 

o 

ERIC 


QUESTiGN  #  :  034 
DATABASE  #   :  013 


♦Accession  Number  Overlap  between  pairs  of  Searchers  * 


Search  Evaiaation 


A: 
B: 
C: 
D: 
E: 
Fs 


#  Relevant  Ql : 

#  Partial ly  Relevant  Q2: 


#  Not 
Total 

#  Not 
Total 


Relevant 

#  Evaluated 
Evaluated 

#  Retrieved 


HI  : 
H2: 
li  : 
12: 
Jl  : 
J2: 


Proportions  *** 

F  in  overlap  between  S1§S2  /  F 

F  in  overlap  between  S1&S2  /  F 

b  in  overlap  between  Sia.S2  /  D 

b  in  over lap  between  S1&S2  /  D 

A+B  in  overlap  between  S1SS2  / 

A+B  in  overlap  between  S1SS2  / 

E  in  overlap  between  S1&S2  /  E 

E  in  overlap  between  S1&S2  /  E 


£  o  r 
for 
for 
for 
A+B 
A+B 
for  SI 
for  S2 


SI 
S2 
SI 
S2 
for 
for 


SI 
S2 


total   for  Question  : 


Searcher 

A 

B 

e 

D 

E 

F 

□Gl 

i 

2 

□ 

3 

0 

3 

006 

•12 

7U 

86 

186 

272 

□17 

□ 

4 

5 

9 

1 

!□ 

□2? 

1 

13 

7 

21 

□ 

21 

□32 

□ 

1 

□ 

1 

□ 

1 

Searcher  Overlap 


Searcher 


SI 

S2 

A 

B 

c 

E 

F 

Gl 

G2 

HI 

H2 

11 

12 

Jl 

□01 

□□S 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

.□□□ 

□ 

.□□□ 

□ 

.□□□ 

□  . 

□□□ 

□  . 

□□□ 

g. 

□□□ 

g. 

ggg 

g. 

0 

□01 

□17 

□ 

i 

□ 

1 

□ 

1 

□ 

.  333 

0 

□ 

.  333 

□  . 

111 

□  . 

333 

□  . 

25  □ 

g. 

ggg 

g. 

g 

□□1 

□29 

□ 

1 

0 

1 

□ 

1 

0 

^  333 

□ 

.□47 

0 

•  333 

□  . 

□47 

□  . 

333 

□ .  071 

□  . 

□□□ 

□  . 

□ 

□□1 

□32 

□ 

□ 

□ 

0 

□ 

□ 

iODD 

□ 

.□□□ 

□ 

.□□□ 

0. 

□□0 

□  . 

□□□ 

□  . 

OO^ 

□  . 

□□□ 

□  . 

□ 

□08 

□17 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

.COD 

□ 

.□□□ 

□ 

.□□□ 

□  . 

□□□ 

□  . 

□□□ 

□  . 

ODD 

□  . 

□□□ 

□  . 

□ 

□□6 

□29 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

-□□□ 

G 

.□□6 

0 

.□□□ 

□  . 

□□□ 

□  . 

□□□ 

□  . 

DOD 

□  - 

□□□ 

□  . 

□1 

DOB 

□32 

□ 

□ 

□ 

g 

□ 

□ 

□ 

.□□□ 

□ 

-□□□ 

□ 

.□□□ 

□  . 

□□□ 

□  . 

□□□ 

□  . 

DDD 

□□□ 

□  . 

□i 

□17 

□29 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

.□□□ 

□ 

.□□□ 

□ 

.□□□ 

□  . 

□□□ 

0. 

□□□ 

□  . 

DDD 

□  . 

□□□ 

0. 

□1 

□17 

□32 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

.□□□ 

□ 

.□go 

□ 

.gg^ 

□  . 

□□□ 

□  - 

□□□ 

□  . 

DOD 

□  . 

□09 

□  - 

□I 

□29 

□32 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

.□□□ 

□ 

.□□□ 

□ 

.□□□ 

□  . 

□□□ 

□  . 

□□□ 

□  . 

000 

□  . 

□□□ 

□  . 

□1 

ERIC 


QUESTION  #  :  035 
DATABASE  #   :  154 


*     Accession  NumkDer  Overlap  kDetween  pairs  of  Searchers  * 
Search  Evaluation  : 


ProporticDns  *** 

A: 

#  Relevant 

Gl  : 

F  in  overlap  between  SI  a.S2  / 

F 

for 

SI 

B: 

#  Partially  Relevant 

G2: 

F  in  overlap  between  Si &S2  / 

F 

for 

S2 

C: 

#  Not  Relevant 

HI  : 

D  in  overl ap  between  si  &S2  / 

D 

for 

Si 

D: 

Total  #  Evaluated 

H2: 

D  in  over 1 ap  between  SI &S2  / 

D 

for 

S2 

E: 

#  Not  Evaluated 

11  : 

A+B  in  overlap  between  S1&S2 

/ 

A+B 

for 

Total   #  Retrieved 

12: 

A+B  in  overlap  between  Si&S2 

/ 

A+B 

for 

Jl  : 

E  iri  bvf=:  w*l  ap  between  SI  &S2  / 

E 

for 

SI 

J2: 

E  in  ovf^r.l  ap  between  SI  &S2  / 

£ 

for 

S2 

Total   for  Question  : 


Searcher 

A 

B 

e 

b 

E 

F 

□01 

1 

g 

g 

1 

g 

1 

008 

o 

5 

5 

16 

0 

16 

017 

1 

0 

0 

1 

g 

1 

029 

12 

4 

6 

22 

0 

22 

032 

9 

ICD 

3 

22 

0 

22 

Searcher  Overlap  : 
Searcher 


SI 

S2 

A 

B 

e 

b 

E 

001 

gos 

0 

0 

g 

g 

0 

001 

017 

1 

g 

g 

1 

g 

001 

029 

0 

0 

g 

g 

g 

001 

032 

0 

0 

0 

g 

Q 

008 

017 

0 

0 

0 

0 

0 

008 

029 

4 

0 

0 

4 

0 

008 

032 

1 

4 

0 

017 

029 

0 

b 

0 

b 

0 

gi7 

032 

0 

0 

0 

0 

0 

029 

032 

1 

1 

0 

2 

0 

F         Gl  G2         HI  H2  II  12         Jl  J 

Q  g.QOb  □.□00  □.□□□  □.OOG  □.□□□  □.□□□  □.□□□  □.□□ 

1  i . sgg  i . □□□  i . □□□  i . □□□  1 , □□□  1 . □□□  □ . □□□  □ , □□ 
g  g.ggg  g.ogg  □.□□□  □.□□□  □.□□□  □.□□□  □.□□□  □.□□ 

□  □.□□□  g.ggg  □.□□□  g.oo^  □.□□□  □.□□□  □.□□□  □.□□ 
g  g.ggg  g.ogo  □.□gg  g.^o^  □.□□□  □.□□□  □.□□□  □.□□ 
4  g.^sg  g.isi  g.asg  g.isi  g.363  g.25Q  □.□□□  □.□□ 
4  g.^sg  □.181  g.25g  g.isi  □.272  g.i57  Q.ogg  □.□□ 

□  □.□□□  □.□□□  □.□□□  g.ggg  g.ggg  g.ggg  g.ggg  □.□□ 

□  □.□□o  □.□□□  □. □□□  □.□□□  g.ggg  g.^gg  o.ggg  o.gg 

2  □.□9^  □.□9^  □.□90  □.□9^  □.ll'B   □.l^B  □.□□□  □.□□ 


EKLC 


459 


QUESTION  #  :  036 
DATABASE  #   :  090 


*     Accession  Number  Overlap  between  jDairs  of  Searchers  * 
Search  Evaluation  : 


.Proportions  *** 

A: 

#  Relevant 

m : 

F  in  overlap  between 

Si&S2  / 

F 

for 

SI 

B: 

#  Partial ly  Relevant 

G2: 

F  in  overlap  between 

S1&S2  / 

F 

for 

S2 

C: 

#  Not  Relevant 

HI  : 

D  in  overlap  between 

S1&S2  / 

D 

for 

Si 

Dt 

Total  #  Evaluated 

H2: 

P  in  overlap  between 

S1&S2  / 

D 

for 

S2 

£: 

#  Not  Evaluated 

11  : 

A+B  in  overlap  betwee 

?n  S15S2 

/ 

A+B 

for 

F: 

Tota 1  #  Retrieved 

12: 

A+B  in  overlap  between  SI 5S2 

/ 

A+B 

for 

Jl : 

E  in  overlap  between 

S15S2  / 

E 

for 

SI 

J2: 

E  in  overlap  between 

S1&S2  / 

E 

for 

S2 

Total   for  Question  : 


Searcher       A  B 


007 

13 

13 

012 

5 

2 

023 

20 

17 

02S 

31 

13 

036 

28 

9 

C 

D 

E 

F 

11 

37 

36 

73 

5 

12 

0 

12 

13 

50 

12 

62 

12 

56 

0 

56 

10 

47 

0 

47 

Searcher  Overlap 
Searcher 


SI 

S2 

A 

B 

C 

D 

E 

F 

Gl 

62 

HI 

H2 

ii 

12 

Jl 

007 

012 

1 

0 

0 

1 

0 

•1 

0. 

□13 

0 

.083 

0 

.027 

0. 

083 

0 

i038 

0 

.142 

QiQOO 

0 

iO 

007 

023 

0 

d 

0 

0 

0 

0 

0, 

000 

0 

.000 

0 

.000 

0. 

000 

0 

-000 

0 

.000 

Q.QOQ 

0 

iO 

007 

025 

2 

0 

1 

3 

d 

3 

0- 

d41 

0,053 

0 

.081 

0. 

053 

0 

.076 

0 

.045 

0.000 

0 

.0 

007 

036 

2 

0 

1 

3 

0 

3 

d- 

d41 

0 

.063 

0 

.081 

0. 

063 

0 

.076 

0 

.054 

0.000 

0 

.0 

012 

023 

2 

0 

0 

2 

0 

2 

0- 

166 

0 

.032 

0 

.166 

0. 

040 

0 

.285 

0 

.054 

0.000 

0 

.01 

012 

025 

4 

2 

1 

7 

0 

7 

0- 

583 

C 

.125 

d 

.583 

0. 

125 

0 

.857 

0 

.136 

0.000 

0 

.01 

012 

036 

4 

2 

i 

7 

d 

7 

0. 

583 

0 

.148 

0 

.583 

0. 

148 

0 

.857 

0 

„162 

0.000 

0 

.01 

023 

D2S 

9 

5 

0 

14 

0 

14 

0. 

225 

0 

.250 

0 

i280 

0. 

250 

0 

.378 

0 

.318 

0.000 

0 

.0( 

023 

036 

9 

S 

0 

14 

0 

14 

0; 

225 

0 

.297 

0 

;28Q 

0. 

297 

0 

.378 

0 

.378 

0 . 000 

0 

.0( 

025 

036 

28 

8 

10 

46 

Q 

46 

0. 

821 

0 

.978 

0 

.821 

0. 

978 

0 

.818 

0 

.972 

0.000 

0 

.d( 

460 


QUESTION  #  :  037 
DATABASE  #.  :  016 


************************* 

*  Accession  Number  Overlap  between  pairs  of  Searchers  * 
**************************************^t^t^t^t^t^t^t^t^t^t^t^t^t^(.^t^t^t^(.^^ 

Search  Evaiciation  : 


*** 

Proportions  *** 

A: 

#  Relevant 

Q  'l  : 

F  in  overlap  between  S15S2  / 

F 

for 

SI 

B: 

#  Partially  Relevant 

<S2: 

F  in  overlap  between  S1&S2  / 

F 

for 

S2 

C: 

#  Not  Relevant 

Hi  : 

D  in  overlap  between  S1SS2  / 

D 

for 

SI 

D: 

Total  #  Evaluated 

H2: 

D  in  overlap  between  S1SS2  / 

D 

for 

S2 

E: 

#  Not  Evaluated 

11  : 

A+B  in  overlap  between  S1&S2 

/ 

A+B 

for 

F: 

Total  #  Retrieved 

12: 

A+B  in  overlap  between  S1&S2 

/ 

A+B 

for 

Jl  : 

E  in  overlap  between  SiaS2  / 

E 

for 

SI 

J2: 

E  in  overlap  between  SI &S2  / 

E 

for 

S2 

Tota^  for  Qaestio 
Searcher       A  B 


□15 

11 

3 

□30 

3 

□ 

□34 

16 

1 

□35 

29 

1 

□41 

31 

6 

n  : 


c 

D 

E 

F 

33 

47 

25 

72 

2 

5 

□ 

5 

6 

23 

2 

25 

6 

36 

19 

55 

1^ 

47 

26 

73 

Searcher 


SI 

S2 

A 

B 

e 

D 

E 

F 

61 

G2 

HI 

H2 

11 

12 

Jl 

J: 

□15 

030 

i 

0 

d 

i 

0 

i 

□ 

.013 

0 . 200 

0. 

021 

0- 

200 

0 

.071 

0. 

333 

-^.ggg 

0. 

ooi 

□15 

034 

0 

d 

□ 

d 

0 

b 

0 

.000 

0.000 

0. 

000 

0. 

000 

0 

,000 

0. 

000 

c  .000 

0. 

ggi 

gis 

035 

0 

0 

0 

d 

0 

0 

0 

.000 

0.000 

0. 

□00 

0. 

000 

0 

.000 

0. 

000 

0.000 

p. 

gg; 

□15 

041 

g 

D 

0 

d 

3 

d 

0 

-GOO 

0- 0.0.9 

0. 

000 

0. 

000 

0 

.000 

0. 

000 

0.000 

0. 

00( 

□30 

034 

0 

g 

g 

0 

0 

d 

d 

.odd 

d'.ddd 

d. 

000 

d. 

doo 

0 

.000 

0. 

000 

0.000 

0. 

00c 

□3^ 

035 

0 

0 

g 

d 

0 

0 

d 

.ddd 

d.dod 

d. 

ddd 

0. 

ddo 

0 

.000 

0. 

000 

0.000 

0. 

00c 

□3^ 

041 

0 

0 

0 

0 

g 

0 

d 

.000 

0.000 

0. 

000 

0. 

000 

0 

.000 

0. 

000 

0.000 

0. 

00c 

□34 

035 

9 

0 

5 

14 

2 

16 

d 

.64d 

0.29d 

0. 

60S 

0. 

388 

0 

.529 

0. 

300 

1 .000 

0. 

lOE 

034 

041 

9 

d 

2 

11 

0 

11 

0 

.440 

0.150 

d. 

478 

d. 

234 

D 

.529 

d. 

243 

0,000 

0. 

ooe 

□35 

041 

15 

0 

17 

1 

18 

0 

.327 

0 . 246 

0. 

472 

0. 

361 

G 

-5G0 

0. 

405 

0.052 

0. 

03£ 

QUESTION  #  :  d3S 
DATABASE  #    :  £361 


************ 

♦Accession  Number  Overlap  between  pairs  of  Searchers  * 
******************************************^t^t^t^t^t^t^^t^(.^(.^t^t^^^ 


Search  Evaiaation 


A:   #  Relevant 

B:   #  Partisklly  Relevant 

C:   #  Not  Relevant 

D:  Total  #  Evaluated 

E:  »  Not  Evaluated 

F:   Total  #  Retrieved 


***     Proportions     ***    _  . 

5  F  in  overlap  betweien  S1SS2  /  F  f  or  SI 
S2:  F  in  overlap  bfcrtv.;^3n  S1&S2  /  F="  for  S2 
Hi:  b  in  overlap  between  SI 8.32  /  D  for  SI 
H2:  D  in  overlap  between  S1SS2  /  D  for  S2 
il : :  A+B  in  over 1 ap  between  S1SS2  /  A+B  for  SI 
12:  A+B  in  over lap  between  S1&S2  /  A+B  for  S2 
Jl :  E  in  overlap  between  S1&S2  /  E  for  SI 
J2:   E  in  overlap  between  S1&S2  /  E  for  S2 


Total   for  Question  : 


Searcher 

A 

B 

C 

D 

E 

F 

□15 

8 

5 

3 

16 

53 

69 

030 

6 

9 

19 

34 

240 

274 

034 

8 

1 

4 

13 

32 

45 

035 

6 

2 

2 

10 

15 

25 

041 

29 

8 

5 

40 

^64 

104 

Searcher  Overlap 


Searcher 


SI 

S2 

A 

B 

c 

D 

E 

015 

030 

0 

d 

0 

d 

d 

D15 

034 

□ 

0 

0 

d 

d 

015 

035 

0 

0 

D 

0 

0 

015 

041 

2 

1 

0 

3 

4 

030 

034 

0 

0 

0 

0 

0 

030 

035 

0 

0 

0 

0 

0 

030 

041 

b 

d 

0 

0 

0 

034 

035 

0 

d 

d 

0 

0 

034 

041 

d 

0 

b 

0 

d 

035 

041 

0 

0 

0 

0 

0 

Gl 


HI 


Hi 


II 


Jl 


0  0.000  0.000  0-000  0-000  0.000  0.000  0.000  g.i 

d  d-ddd  0.000  o.ooo  o.ooo  o.ooo  o.boo  o.ooo  o.i 
0  d.ooo  o.ooo  d.doo  d.ooo  o.ooo  o.ooo  o.ooo  o.i 

7  0.101   0-067  0-187  d.d75  0.230  0.081   0.075  O.I 

0  D.OOO  o.Odo  o.ddd  d.ddd  d.ddd  o.ooo  o.ooo  o.i 

0  o.DOD  0.000  D.ooo  0.000  O.ddd  d.ddd  d.ooo  o.i 

0  0.000  0.000  0.000  D.ooo  o.ddo  d.ddd  d.ddd  d.i 

0  0-000  Q-Qpg  0-000  o-DOD  D-noo  0-000  d.ddd  d-i 

0  0-000  0-000  0-000  P-OOO  O-ODO  D-ODD  D-DOO  0-( 
0  0-000  b.OOO  0.000  o.ooo  0.000  0.000  0.000  0.( 


EKLC 


4B2 


QUESTION  #  5  039 
DATABASE  #    :  015 


************************** 

*  Accession  Number  Overlap  between  pairs  of  Searchers  * 
***************** 

Search  Evaluation  : 


-^-^-^ 

Proportions  *** 

A: 

#  Relevant 

Gl  : 

F  in  overlap  between  Si&S2  / 

F 

for 

SI 

B: 

#  _Partially  Relevant 

G2: 

F  in  overlap  between  S1&S2  / 

F 

for 

S2 

C: 

#  Not  Re 1 evant 

HI  : 

P  in  overlap  between  S1&S2  / 

D 

for 

SI 

b: 

Total  #  Evaluated 

H2: 

D  in  overlap  between  S15S2_/ 

D 

for 

S2 

E: 

#  Not  Evaluated 

11  : 

A+B  in  overlap  between  SI SS2 

/ 

A+B 

for 

Si 

F: 

Total   #  Retrieved 

12: 

A+B  in  overlap  between  S1&S2 

/ 

A+B 

for 

S2 

Jl  : 

E  in  overl ap  between  SI SS2  / 

E 

for 

SI 

J2: 

E  in  overlap  between  SI &S2  / 

E 

for 

S2 

Total   for  Question  : 


Searcher 

A 

B 

C 

D 

E 

F 

□15 

□ 

□ 

□ 

□ 

1 

1 

□30 

4 

3 

7 

14 

□ 

14 

□34 

2 

5 

5 

12 

□ 

12 

□35 

0 

□ 

6 

6 

□ 

6 

□41 

i 

□ 

1 

2 

□ 

2 

Searcher  Overlap  : 


Searcher 


SI 

S2 

A 

B 

C 

D 

E 

F 

Gl 

G2 

Hi 

H2 

11 

12 

Ji 

015 

□30 

□ 

□ 

g 

g 

g 

g 

g. 

ggg 

g. 

000 

0 

.000 

o.oog 

0 

.000 

g. 

ggg 

g. 

ogg 

Oil 

□15 

CD34 

□ 

□ 

□ 

□ 

□ 

□ 

g. 

□□□ 

0. 

ggg 

0 

-ogg 

D.ogg 

g 

-gog 

0. 

000 

0. 

000 

0.1 

□15 

□35 

□ 

□ 

□ 

0 

□ 

□ 

□  . 

□□□ 

g. 

ggg 

g 

.000 

0.000 

0 

.ggg 

0. 

000 

g. 

goo 

g.i 

□15 

□41 

□ 

□ 

□ 

□ 

□ 

□ 

□  . 

□□□ 

g.ggg 

0 

.000 

0.000 

0 

.000 

0. 

000 

g. 

ggg 

0.1 

Q30 

□34 

□ 

0 

□ 

□ 

□ 

□ 

□  . 

□□0 

0. 

000 

0 

.000 

o.ood 

0 

.000 

0. 

000 

0. 

000 

g.! 

Q3D 

□35 

□ 

□ 

□ 

□ 

□ 

□ 

□  . 

□□□ 

0. 

000 

0 

.000 

0 . 000 

0 

.000 

0. 

000 

0. 

000 

0.1 

030 

□41 

□ 

□ 

□ 

□ 

□ 

□ 

□  . 

□00 

0. 

000 

0 

.000 

0.000 

0 

.000 

0. 

000 

0. 

000 

OA 

□34 

035 

0 

□ 

□ 

□ 

□ 

□ 

0. 

□□□ 

0. 

000 

0 

.000 

0.000 

0 

.000 

0. 

000 

0. 

000 

OA 

□34 

041 

□ 

□ 

□ 

□ 

□ 

□ 

□  . 

□□□ 

0. 

000 

0 

.000 

0.000 

0 

.000 

0. 

000 

0. 

000 

OA 

□35 

□41 

□ 

□ 

□ 

□ 

□ 

0 

□  . 

□00 

0. 

000 

0 

.000 

0.000 

0 

.000 

0. 

000 

0. 

000 

OA 

W3 


QOESTION  #  s  040 
DATABASE  #   :  016 


*     Accession  Number  Overlap  between  pairs  of  Searchers  * 


Search  Evaluation  s 


A:  #  Relevant 

B:  #  Partially  Relevant 

Cs  #  Not  Relevant  

D:  Total  #  Evaluated 

E:  #  Not  Evaluated  

F:  Total  #  Retrieved 


***     Proportions  ***   

Gl :  F  in  overlap  between  S15S2  /  F  for  SI 

Q2:  F  in  overlap  between  S1SS2  /  F  for  S2 

HI :  b  in  overlap  between  S1SS2  /  D  for  SI 

H2:  b  in  over lap  between  S1SS2  /   D  for  S2 

ii :  A+B  in  overlap  between  SI  SS2  /  A-4-B  for  SI 

12:  A+B  in  over! ap  between  S1&S2  /  A-4-B  for  S2 

Ji :  E  in  overlap  between  S1&S2  /  E  for  SI 

J2:  E  in  overlap  between  S1&S2  /  E  for  S2 


Total  for 

Que 

stibn 

Searcher 

A 

B 

C 

D 

E 

F 

015 

10 

6 

0 

16 

61 

77 

030 

2 

4 

32 

38 

4 

42 

634 

6 

2 

0 

S 

7 

15 

035 

11 

4 

0 

15 

47 

041 

16 

9 

e 

25 

is 

Searcher  Overlap 


Searcher 


SI 

S2 

A 

B 

c 

D 

E 

F 

Gl 

Q2 

HI 

H2 

11 
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APPENDIX  E.  GROUPS  OF  QUESTIONS  SEARCHED  BY  SETS  OF  OUTSIDE  SEARCHERS 


The  40  queistionis  can  be  grouped  by  type  of  question  as  follows: 
Group  1  -  Medical 

Group  2  -  Psychology  and  Socidldgy 

Group  3  -  Business  Management 

Group  4  -  Scientific  and  Technical 

iSroup  5  -  ®^siness  and  Science 

Group  5  -  Humanities  and  Social  Science 

Group  7  -  Miscellaneous 


For  each  group  of  questions,  the  searcher  numbers  of  the  five 
outside  searchers  who  searched  the  question  are  noted.  This 
compilation  shows  the  specific  questions  searched  in  common  by  the 
various  individual  searchers. 


4^5 


Appendlx-E 


GROUPS  OF  QUESTIONS  SEARCHED  BY  JETo  OF  OUTSIDE  SEARCHERS 


Group  1  -  Medical  Questions 


Searcher 

Number  Question  Number 


Q004 

Q008 

QOlO 

QOll 

Q015 

S009 

X 

X 

X 

X 

X 

SQiO 

X 

X 

X 

X 

X 

S024 

X 

X 

X 

X 

Sd28 

X 

X 

X 

X 

X 

S037 

X 

X 

X 

X 

X 

Group  5;  - 

Psychology  and  Sociology  Questions 

Saarcher 

Number 

Question  Number 

QOOl 

Q003 

0009 

Q016 

Q021 

Q025 

S002 

X 

X 

X 

X 

X 

X 

S005 

X 

X 

S013 

X 

X 

SOU 

X 

X 

X 

X 

S016 

X 

X 

X 

X 

X 

X 

S021 

X 

X 

X 

X 

X 

X 

S033 

X 

X 

X 

X 

Group  3  -  Business  Management  Questions 


Searcher 

Number  Question  Number 


QO02 

Q005 

Q007 

Q013 

Q014 

Q018 

S003 

X 

X 

X 

X 

X 

X 

S004 

X 

X 

X 

X 

X 

X 

S006 

X 

X 

X 

X 

Y 

Y 

A, 

S013 

X 

X 

X 

X 

S026 

X 

X 

X 

X 

X 

X 

Sb33 

X 

X 

Group  4  - 

Scientific 

and  Technical  Questions 

Searcher 

Number 

Question 

Number 

Q006 

Q012 

Q017 

Q022 

Q027 

Q029 

S005 

•X 

X 

S008 

X 

soil 

X 

X 

X 

X 

X 

S025 

X 

S027 

X 

X 

X 

X 

X 

X  . 

S029 

X 

S032 

X 

S038 

X 

X 

X 

X 

S039 

X 

X 

X 

X 

S040 

X 

X 

X 

X 

X 
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Group  5  -  Business  and  Science  Questions 
Searcher 


Number 

Question  Nomber 

Q019 

Q020 

Q032 

Q033 

Q034 

Q035 

SOOl 

X 

X 

X 

X 

X 

X 

S005 

X 

X 

S008 

X 

X 

X 

X 

X 

S013 

X 

X 

sei7 

X 

X 

X 

X 

X 

X 

3829 

X 

X 

X 

X 

S032 

X 

X 

X 

X 

X 

Group  6  - 

Humanities 

and  Social 

Science 

Questions 

O  XZCL  I.  L^ilCf  JL 

^iirhTs  AT* 

it  UUl  Ucf  JL 

Question  Number 

Q024 

Q026 

Q028 

Q030 

Q031 

Q036 

S007 

X 

X 

X 

X 

X 

S012 

X 

X 

X 

X 

X 

X 

S015 

X 

S023 

X 

X 

X 

X 

X 

S025 

X 

X 

X 

X 

S034 

X 

S036 

y 

X 

X 

X 

X 

Group  7  - 

Miscellaneous  Questions 

Searcher 

Number 

Question  Number 

Q023 

Q037 

Q038 

Q039 

Q040 

S015 

X 

X 

X 

X 

X 

S030 

X 

X 

X 

X 

X 

3834 

X 

X 

X 

X 

X 

3635 

X 

X 

X 

X 

X 

S04t 

X 

X 

X 

X 

X 

APPENDIX  F.  FLOWCHARTS  FO^l  OVERALL  DESIGN  OF  THE  PROJECT 


F-1  -  LIST  OF  EXPERIMENTS 

List  of  the  experiments  for  the  overall  design  of  the  project  and 
referenced  in  subsequest  flowcharts 


F-2  -  OVERALL  DESIGN  OF  THE  PROJECT  -  FLOWCHART  1 

Flowchart  showing  the  major  steps  in  conducting  the  experiments  for 
the  project 


F-3      DESIGN  OF  THE  PROJECT  FOR  QUESTION  ANALYSIS  -  FLOWCHART  2 

Flowchart  showing  the  major  steps  in  question  analysis  and 
classification 


F-4      DESIGN  OF  THE  PROJECT  FOR  USER  QUESTIONS  FOR  ONLIt^E  SEARCHING  - 
FLOWCHART  3 


Flowchart  showing  the  major  steps  for  conducting  the  360  onlire 
searches  for  the  40  questions 


F-5      OUESTTON  ANALYSIS  CIIART  -  FLOWCHART  4 

Flowchart  showing  the  major  steps  for  question  analysis  and 
classification 
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NSF   PROJECT  IST-S5D5411 

EX_PERiriENtS_iN  THE  COGN  OP 
INFORMATION  SEEKING  AND  INFORMATION  RETRIEVING 

LIST  OF  EXPERIMENTS 

TITLE 

VALIDATION  OF   CONTEXT  ELEMENTS 
EFFECTS  OF  CONTEXT 
REQUEST  STRUCTURE 
REQUEST  CLASSIFICATf ON 
EFFECTS  OF  REQUEST  CLASSES 

NATUPE  OF   REQUEST  ANAllYSIS  AND  SEARCH  STRATEGY  PROCESS 
ELABORATION  OF  CONCEPTS 
SEARCH  TACTIC5 

AGREEMENT  ON  SEARCH  STATEMENTS 
tdGNITIVE  CHARACTERISTICS 

COGNITIVE   CHARACTERISTICS  AND  PERFORMANCE 
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NSF  PROJECT  IST-S505411 
EXPERiMENtS  IN  THE  COGNITIVE  ASi=ECTS  OF 
INFORMATION  SEEKING  AND  INFORMATION  RETRIEVING 

OVERAbb  DESIGN  OF     PROJECT  -     FLOWCHART  1 
OBTAIN  QUESTiOVS  FOR  THE  EXPERIMENTS 

i 

.  i 

CONTACT  LIBRARIES  THAT   PERFORM  ONLINE  SEARCHES 

i 

I 

SELECT  160  QUESTIONS   (GROUP  1 )    FOR  WHICH  ONLINE  SEARCHES  HAVE  BEEN  COMPLETED 

I 

:  I 
ANAL'.  ;..     :.:  iUEST  STRUCTURE  AND   CLASSIFICATION    (EXP.   3,   4;)    FOR  GROUP  i  (Fll) 

I 
! 

CONTACT  USERS  WITH  QUESTIONS  NEEDING  ONLINE  SEARCHES  CFl) 

I 
I 

SELECT  40  USER  QUESTIONS   (GROUP  2.')   FOR  EXPERIMENTS  ON  ONLINE  SEARCHING 

I 
I 

CONTACT  SEARCHERS  TO   CONDUCT  ONLINE  SEARCHES  FOR  SROUP  2  (F2-3;C1^ 

I 
I 

SELECT  38  E-EARCHERS  FOR  ONLINE  SEARCHES     FOR  GROUP  2 

I 
I 

DETERMINE  THE  COGNITIVE  SCORES  OF  EACH  ONLINE  SEARCHER   (EXP.  rtP) 

I 

i 

OBTAIN  WRITTEN  QUESTIONS     FOR  GROUP  2    CF4  , 5  ;  Pi  ,  e2.'> 

! 
I 

AfJALYZE   REQUEST  STRUCTURE  AND   CLASSIFICATION  FOR  GROUP  2    CEXP.    3,  4)(:f11.') 

I 

1 

CONDUCT  AND  TAPE  USER  INTERVIEWS  FOR  GROUP  T  QUESTIONS  CF6-9,P2) 
DFTERMINE  THE  CONTEXT  VALUES  FOR  EACH  USER  AND  SEARCHER   CEXP.   1  .•) 

I 

1 

con:  -oO  online  searches  using  different  SEARCH  STRATEGIES   (:9  DIFFERENT 

bL,.ixCHES  FOR  EACH  GROUP  2  QUESTION.)      CEXP.    2,    7,    1 1 .1  CFlO ,  1 3 ,  P3-S) 

I 

I   

MERGE   RESULTS  FROM  ALL  ONLINE  SEARCHES  OF  GROUP  2  QUESTIONS 

I 

1 

SEND  SEARCH  HiESULTS  FOR  GROUP  2  TO  USERS  FOR 
EVALUATION  OF  EFFECTIVENESS  CF12,14.1 

I 

to   PAGE  2  OF  PROJECT  FLOWCHART 


<EXP.=  EXPERIMENT  #;    C  =   CODE  FORMR  «s   r       FORM  #;    P  =   PROCEDURE  #: 


ERIC 


OVERALL  DESIGN  OF     PROJECT  -     FLOWCHART   (Page  2) 


FROM  PAGE  1   OF  PROJECT  FLOWCHART  1 

I 

;  •  I  _ 

DETERMINE  THE     EFFECTIVENESS  OF  EACH  OF  THE 
360  SEARCHES  OF  GROUP  2  QUESTIONS 

I 

i 

DETERMINE  tHE  EFFICIENCY  OF  THE 
36d  SEARCHES  OF  GROUP  2  QUESTIONS 

I 
I 

DETERMINE  THE  SEARCH  TACTICS  USED  FOR 
THE  360  SEARCHES  OF  GROUP  2  QUESTIONS    CEXP.  8> 

i 
i 

DETERMINE  THE  EFFECT  OF   CONTEXT  VALUES  ON  EFFECTIVENESS 
AND  EFFICIENCY    <EXP.    2)    FOR  ALL  GROUP  2  QUESTIONS 

I 
I 

DETERMINE  EFFECTS  OF   REQUEST   CLASSIFICATION  ON  EFFECTIVENESS 
AND  EFFICIENCY   CEXP.    5)    FOR  ALL  GROUP  2  QUESTIONS 

I 
I 

DETERMINE  EFFECTS  OF  4  SEARCH  STRATEGIES  BASED  ON   CONCEPT  ELABORATION 
ON  EFFECTIVENESS  AND  EFFICIENCY    CEXP.    7)    FOR  ALL  GROUP  2  QUESTIONS 

I 
I 

DETERMINE  EFFECTS  OF  DIFFERENT  SEARCH  TACTICS  ON 
EFFECTIVENESS  AND  EFFICIENCY    CEXP.    8.)    FOR  ALL  GROUP  2  QUESTIONS 

i 

i 

DETERMINE  EFFECTS  OF   COGNITIVE  SCORES  ON  EFFECTIVENESS 
AND  EFFICIENCY    CEXP.    11 .)    FOR  ALL  GROUP  2  QUESTIONS 

J 

I 

SELECT  5  QUESTIONS  FROM  GRt;  "     .    C GROUP  2A:)    FOR  FURTHER  ANALYSIS 


CONDUCT  ONLINE  SEARCHES  AND  TAPE  THE  SEARCH  PROTOCOLS 
FOR  OUTSIDE  SEARCHERS  FOR  GROUP  2A  QUESTIONS 

I 
I 

DETERMINE  THE   PROCESSES  USED   IN   REQUEST  ANALYSIS  AND 
SEARCH  STRATEGY  FORMULATION    CEXP.    6)    FOR  GROUP  2A  QUESTIONS 

I 

i 

ADD  4  QUESTIONS    CSROUP  3.)    FROM  DR.    WOELFL  DISSERTATION 

I 
I 

DETERMINE  THE  DEGREE  OF  AGREEMENT  ON 
SEARCH  STATEMENTS  FOR  GROUPS  2  AND  3  QUESTIONS   (EXT.  9.) 

I 
I 

__ DETERMINE  THE  EFFECT  OF   COGNITIVE  SCORES  ON  THE  DEGREE 
OF  AGREEMENT  VALUES  FOR  GROUPS  2  AND  3  QUESTIONS  CEXP. 
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NSF  PROJEet  iSt-S5Q54li 
EXPERIMENTS  IN  THE  eesNiTiVE  ASPECTS  OF 
INFORMATION  SEEKING  AND  INFORHATION  RETRIEVING 


OVERALL  DESIGN  OF     PROJEeT  -  FLOWCHART 

DESIGN  OF  PROJECT  FOR  QUESTION  ANALYSIS  FLOWCHART  #2 

OBTAIN  QUESTIONS  FOR  THE  EXPERIMENTS 

I 

  i   

CONTACT  LIBRARIES  THAT   PERFORM  ONLINE  SEARCHES 

I 

i  

SOURCES  USED  WERE: 
UNIVERSITY  OF  DAYTON 
CWRU  MEDICAL  LIBRARY 
AMERICAN  SOCIETY  FOR  METALS 
CHAMIS  DISSERTATION  MATERIAL 
WOELFL  DISSERTATION  MATERIAL 
I 

i 

SELECT  160  QUESTIONS    (GROUP  1)    FOR  WHICH  ONLINE  SEARCHES  HAVE  BEEN   COMP'  ETt 

I 

1  

SAMPLE  QUESTIONS 
#38  The  effects  of   aging  on  person's  self   image  and  ability 
to  cope  w.'th  the  procRss   Cage  group  -  elderly) 

I 

i 

ANALYZE  REQUEST  STRUCTURE  AND  CLASSIFICATION    CEXP.    3,    A)    FOR  <^"-     "    -  «F1i3 

I 

i 

SAMPLE  QUESTION  ANALYSIS    (FORM  11) 

I 

1 

EXP.   3     REQUEST  STRUCTURE 
ANALYSIS  OF  QUESTION  TO  DETERMINE  THE  PRESENCE 
OF  ONE  OR  MORE  OF  THE  THREE  BASIC  ELEMENTS 

<■^^^    SUBJECT  -  THE  Cr^NTRAL   CONCEPT  OF  THE  SEARCH 

#3Sr  i-'   ig ing J    b-l   person's  self-image;  c.>process 

<2:>   QUERY  -  THE  SPECIFIC  ASPECT  ASKED  ABOUT  THE  SUBJECT 
#38  -a>   effects;    b)   ability  to  cope 

<3)    LEAD-IN  -  PHRASE  PRECEDING  SUBJECT  OR  QUERY  AND  NOT 
DIRECTLY  SEARCHABLE 

#38  -  none  but  examples  are  -  what,  where, 
interested   in,  why 

The  subject  or  the  query  may  sometimes  also  have 
MODIFERS  and/or  CONSTRAINTS 


CD  MODIFIERS  -  MORE  SPECIFIC  OR  ALTERNATIVE  CONCEPTS  FOR  THFi  SUBJECT  OR  QUE 
C2.')    CONSTRAINTS  -  limitations  on  the  type  of   information  provided. 

For  #38  =   1  (current) 

I 

TO  PAGE  2 


ERIC 


DESISN  OF  PROJECT  FOR  QUESTION  ANAL'^SIS  FLOWCHART  #2   (PAQE  2.*) 


FROM  PAGE  •> 
I 
I 

EKP.    4   REQUEST  CLASSIFICA < l6f : 


REQUESTS  ARE   CtASSiFiED  ACCORDING  TO  ^  ATTRIBUTES 

CI)    DOMAIN  -  Number  of   Dial  index  categories  to  which  search  could   be  assignee 
*^3S  -i")   F^yschoiogy;   2)   Social  Science?   3^   Meline;   4)  Scisearch 


€2.*)    CLARITY  -  The  average  value  of   two  scaled  measures,    range   1   to  5 ,   #38  =  ^ 
a")  Semantics  -  the  meaning  of  the  terms       #38  =  5 
b>   Syntax  -  the  relation  between  terms  -  #38  =  4 

(3>   SPECIFICITY  -  The  average  value  of   scaled  measures  for  the  hierarchical 

level   from  general    Cmeta   language)   to  specific  (.'object 
language  :>   for  each  subject  and  query,   ranging  from  1  to  5 . 
For  #38  =3.? 


C4.J    COMPbEXlTY  -  Coniists  of  2  values 

a)   #  search  concepts;   for  #38  =  3 
B)   #  constraints;    for  #38  =  1 

TESTS  will   be  conducted  to  determine  the  degree  of   fit  to  the  model 
and  the  degree  of  agreeement  between  two  analysts 


(Exp,«  Experiment  #;  C^Code  Forms  #;  F=Form  #;  P=  Procedure  #  ) 
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NSF  PROJECT  IST-85DBA11 

EXPERIMENTS  IN  THE  COGNITIVE  ASPECTS  OF  

INFORMATION  SEEKING  AND  INFORMATION  RETRIEVING 


DESIGN  OF  PROJECT  FOR  USER  QUESTIONS  FOR  ONLINE  SEARCHING 

FLOWCHART  #3 


CONTACT  USERS  WITH  QUESTIONS  NEEDING  ONLINE  SEARCHES  <F1.> 

I 
I 

SOURCES  CONTACTED  WEREs 
CAMPUS  NEWS  ARTICLE 
FLYERS  SENT  TO  ALL   CASE  WESTERN  RESERVE 
DEPARTMENTS  WITH  GRADUATE  STUDENTS 
FLYERS  POSTED  ON  MAJOR  BULLETir4  BOARDS  ON  CAMPUS 
FLYERS  SENT  TO  OTHER  CAMPUSES 
AKRON  UNIVERSITY,    CLEVELAND  STATE  UNIVERSITY, 
JOHN  CARROLL  UNIVERSITY,    KENT  STATE  UNIVERSITY 
FLYERS  a  LETTERS  SENT  TO  MEMBERS  OF 
SPECIAL  LIBRARIES  ASSOCIATION,    CLEVELAND  CHAPTER 
NORTHEAST  OHIO  i^MERiCAN  SOCIETY  FOR   INFORMATION  SCIENCE 
CLEVELAND  ONLINE  USERS  GROUP 
I 
I 

SELECT  AO  USER  QUESTIONS   (GROUP  2!)    FOR  EXPERIMENTS  ON  ONLINE  SEARCHING 

I 
I 

CONTACT  SEARCHERS  TO  CONDUCT  ONLINE  SEARCHES  FOR  GROUP  2  CF2-3;C1.') 

I 
I 

SOURCES  CONTACTED  WERE: 
SPECIAL  LIBRARIES  ASSOCIATION,    CLEVELAND  CHAPTER 
NORTHEAST  OHIO  AMERICAN  SOCIETY  FOR  INFORMATION  SCIENCE 
CLEVELAND  ONLINE  USERS  GROUP 
I 
I 

SELECT  38  SEARCHERS  FOR  ONLINE  SEARCHES     FOR  GROUP  2 

i 

i 

DETERMINE  THE  COGNITIVE  SCORES  OF.EACH  ONLINE  SEARtHRR   <EXP.  lO) 

i  " 
I 

THREE  COGNITIVE  TESTS  ARE  USED 
LEARNING  STYLE  INVENTORY 

REMOTE  ASSOCIATES  TEST 
EMPLOYEE  APTI^UTE  SURVEY 
I 

i 

TO   PAGE  2 


(Exp.  =  EXPERir^ffiNT  #;  C=  CODE  FORMS  #;  IF  =  FORM  #;  P=  PROCEDURE  #  ) 


DESISN  OF  PROJEGt  FOR  USER  QUEStiONS  FOIR  ONLINE  SEAIRCHiNG 

FbdWeNART  #3   (page  2> 

OBTAIN  WRITTEN  QUESTIONS     FOR  GROUP  2   <F4  ,5  ;  PI  ,  02!) 

i 

I 

SAMPbE  QUESTIONS  -  QUESTION  #11 
WHAT  ARE  THE  PYSGHOEMOTIONAL  AND  PSYOOSOeiAL:  RESPONSES  OF  PARENTS 

AND  SURVIVING  SIBL-INGS  TO  THE  DEATH  OF  AN  INFANT,    DUE  TO 
SUDDEN  INFANT  DEATH  SYNDROME   (S.I.D.S-;   ALSO  CALLED   "CRIB  DEATH") 
AND  WHAT  ARE  THE  COPING  STRATEGIES  TO  THESS  FAMILY  MEMBERS? 

I 
I 

ANALYZE  REQUEST  STRUCTURE  AND  CLASSIFICATION  FOR  GROUP  2   <EXP.   3,  4>(:^il.) 

I 

 I 

CONpyCT  AND  TAPE  USER  INTERVIEWS  FOR  GROUP  2  QUESTIONS  CF6-^V,P2:) 
DETERMINE  THE  CONTEXT  VALUES  FOR  EACH  USER  AND  SEARCHER    <£XP.  1> 

I 

I_  _ 

_CQNTEXT  VALUES  FOR.QUb      .  #11 
POSSIBLE  VALUES   RANGE,  •  ,^>':.<-   i   TO  S 
PROBLEM  DEF'INITION  SCALE-     USER   "^^    ViARCHERS  -P.S.  4 
.   INTENT  SCALE  -  USER_25   SEAKwilERS-  P.S-  1 
PROBLEM-PUBLIC  KNOWLEDGE  SCALE  -     USER  4?   SEARCHERS.-  P.S.   3;   S,   5,   3,   5,  4 
INTERNAL  KNOWLEDGE   ^'^ALE  -  USER  4;   SEARCHERS  -  P.S.   1;   2,   4,   1,   4,  3 

I 

_  _  I  

CONDUCT  360  ONLINE  SEARCHES  USING  DIFFERENT  SEARCH_STRATEGIES   C9  DIFFERENT 
SEARCHES  FOR  EACH  GROUP  2  QUESTION)      <EXP/2,   7,   11  >  (Fl 0 , 1 3 ,  PS-B.) 

I 

__1 

 9_SEARCHES  FOR     QUESTION  #11 

OUTSIDE  SEARCHERS  009*   024,   02S*   010,  037 
PROJECT  SEARCHES  119,    219,    319,  A19 

! 

I    _  __ 

SEARCH  RESULTS_FOR  QUESTION  #11 
#REFERENCES  -  9,    98,    45,    54,    5,    198,    144,    21,  30 

TOTAL  604 
I 

I  

MERGE  RESULTS  FROM  ALL  ONLINE  SEARCHES  OF  GROUP  2  QUESTIONS 

I 
I 

MERGED  SET  FOR  QUESTION  #11-  310 

I 
I 

SEND  SEARCH  RESULTS  FOR  GROUP  2  TO  USERS  FOR 
EVALUATION  OF  EFFECTIVENESS  (:F12,14) 

I 
I 

150  ABSTRACTS  SLNT  TO  THE  USER  FOR  EVALUATION  FOR  QUESTION  #11 

I 

__      I  _  _ 

EVALUATION   RECEIVED  FOr;  QUESTION  #11 
RELEVANT:^  9;    PARTIALLY  RELE'/ANT  =  16?   NOT   RELEVANT  =  125 

I 

  i 

EVALUATION  COD^G  ENIJIRED  FOR  PROC^^SSl.iG 
BY  COMPUTER  PROGRAMS 
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NSF  PROJECT  iST-SSdSAll 


EXPERIMENTS  IN  THE  COSNITiVE  ASPECTS  OF 
INFORMATION  SEEKING  AND  INi=-ORMATlbN  RETRIEVING 


QUESTION  ANALYSIS  CHART 
FLOWCHART  #4 


160  QUESTIO 
FROM  OUTSIDE 
SOURCES 


40  USER  QUEJi^'IONS 
FROM  REST  OF 
PROJECT 


QUESTION  ANALYSIS 
BY  2  ANALYSTS 
FORM  11 


I 


REQUEST 
STRUCTURE 
rXP.  3 


T 


REQUEST 
CLASSIFICATION 
EXP.  A 


L 


3 


CORRELATE 
RESULTS 


DEGREE  OF  FIT 

STRUCTURE 

CLASSIFICATION 


DEGREE  OF 
AGREEMENT  BETWEEN 
ANALYSTS 


ERIC 
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i^PENblZ  G.  FLOWCHART  FOR  COMPDTER  PROGRAMS  FOR  THE  PROJECT 
e-t      tlSt  OF  PROGRAMS  IN  dPERAflONAL  ORDER 

G->2    Analysis  of  36d  online  searches  -  flowchart  5 
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LIST  OF  COMPUTER  PROGRAMS 
BRIEF  r>ufLINE  OF  PflOGRAMS  IN  OPERATIONAL  ORDER 


1 .  STEP6 

2.  STEPy 

3.  STEPl 

4.  STEP2 

5.  STEPS 

6.  STEPS 

7.  STEP4 

8.  sfEP9 

9.  STEPS 

10.  STEPll 

11.  STEPl O 

12.  STEP12 


Selects  data  from  original   Dialog  transcript 

Separates  selected  data  into  commands  used,  time  spent,  and 
accession  numbers  retrieved  by  the  searcher  for  each  question 
searched 

Creates  and  updates  master  lists  of  questions  and  searchers 

1.  Creates  a  list  of  searchers  of  each  que*5tion 

2.  Sorts  and  merges  accession  numbers 

Prepares  merged  accession  number  file  to  upload  to  Dialog 

Prepares  downloaded  abstract^-^  to  senri  1:0  the  user'  for 
evaluation 

Assigns  relevance  judgements  to  merged  accession  numbers 

Assigns  relevance  judgements  to  individual  searcher's 
accession  number  set 

Adds  offline  time  to  online  time 

Sel eots  search  terms  used 

Prepares  final  data  on  each  searcher 

Prepares  final   data  on  each  question 
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rslSF  PRbJECt  lST-8505411 

 EXPEBIMESTS  IN  THE  GOSNITVE  ASPECTS  OF 

INFORMATION  SEEKING  AND  INFORMATION  RETRIEVINQ 

PROGRAMS  FOR  ANALYSIS  OF  360  ONLINE  SEARCHES  -  FLOWCHART  #5 

Process  the  unedited  search  transcript  file  recorded  on 
the  hard  disk  for  each  searcher  using  STEP6 


Deletes  all  but  command  lines  and  accession  numbers 


Print,   edit  and  correct  any     typing     and  trars<7.t  ssion  errors 


Process  the  edited  search  text,   using  STEP7 


Creates  separate  files  for 
the  commands,   the  connect  time  and  the  accession  numbers 
for  each  question  and  isearcher 

I 

Print,   edit  and  correct  the  accession  numbers 


Create  file  namss  for  storing  data  for  each  question,  using  STEPl 


Process  the  edited  accession  numbers,   using  STEP2 


Creates  jsorted  files  of  accession  numbers 
Sorts  accession  numbers  by  searchers 
Merges  accession  numbers  by  question 


Process  the  merged  accession  numbers  for  each  question,   using  STEPS 


Creates  a  file  of   accession  numbers  with  commands  to  send  to  DIALOG 


Send  file  of  accession  numbers  to  DIALOG  to  print  abstracts 


Process  file  cf  abstracts,   downloaded  on  disk,   using  STEP5 


Inserts  a  line  at  the  end  of  each  abstr&ct  for  user 
to  sel ect  appropriate  relevancy  statement 

I 

Send  abstracts  to  users  for  evaluation 


To  Page  2  of  program  flowchart 
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PR0SRAMS  FOR  ANALYSIS  OF  360  ONLINE  SEARCHES  -  FLOWCHART  #5  . 


From  Page  1  of  program  flowchart 

I 

Evaluated  abstracts  received  from  users 

I 

Process  file  of  abstracts  evaluated  by  user,   using  STEP4 

I 

Shows  each  accession  number  and  asks  for  user's  evaluation  of  relevance 


Creates  a  file  of  the  user's  evaluation  of  relevancy 

I 

Edit  the  relevancy  statements  for  data  entry  errors,   using  STEP9 

  I 

Creates  a  verified  file  of  the  relevancy  statements 

I 

Process  the  connect  time  file,   and  enter  offline  search  time,   using  STEPS 


Creates  a  file  of  the  online  connect  time,  offline  search  preparation  tini 

and  the  total  search  time  for  each  searcher 

I 

Process  the  relevancy  statement  files,   the  connect  time  file, 
and  the  command  file,   usinrj  STEPICD 


Creates  the  final  data .file  for  each  searcher 

Creates  lists  of  che  search  terms  used  by_ 
each  outside  searcher^  using  STEPll 

Creates  the  final^data  file  for  each  question^ 
using  STEP12 
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APPENDIX       FORMS  DSED  FOR  Tffi  PROJECT  (NUMBERED  1  THROUGH  16) 


From 

Ntimbar  Title 


1  "A  Free  Online  Search!" 

2  Searcher  Profile  (Preliminary) 

3  Searcher  Quest tohnaire  (includes  database  and  thesauri  lists) 

4  Sttstructibns  for  Users 

5  Question  Request  Foirm 

6  interview  Notification 

7  ir^erview  Questionnaire 

8  ebntext  Questionnaire  (User) 

9  interview  Evaluation  and  Context  Questionnaire  (Project 
Searcher) 

id  Context  Questionnaire  (Searcher) 

11  Question  Analysis 

12  User  Questionnaire  on  Evaluation  of  Answers 

13  Search  Record  Form 

14  Cover  tetter  to  Users  to  Accompany  Abstracts 

15  User  Questions  for  Online  Searches 

16  Search  Term  Overlap 
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APPENDIX   H  FORMS 

Shimber  f  ttle 

1  "A  Free  Online  Search I" 

2  Searcher  Profile  (Preliminary) 

3  Searcher  Questionnaire  (includes  database  and 
thesauri  lists) 

4  Instructions  for  Users 

5  Question  Request  Form 

6  Interview  Notification 

7  Interview  Questionnaire 

8  Context  Questionnaire  (User) 

9  interview  Evaluation  and  Context  Questionnaire 
(Project  Searcher) 

10  Context  Questionnaire  (Searcher) 

11  Question  Analysis 

12  User  Questionnaire  on  Evaluation  of  Answers 

13  Search  Record  Form 

14  Cover  Letter  to  Users  to  Accompany  Abstracts 

15  User  Questions  for  Online  Searches 

16  Search  Term  Overlap 
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CASE  WESTERN   RESERVE  UNIVERSITY 


_     —  -  Form  i 

CLEVELAND.   OHIO  4410 


A  FREE  ONLINE  SEARCH! 
DO  YOU  HAVE  A  QUESTION  FOR  SEECHING? 


DO  YOU  NEED  A  LITERATURE  SEARCH  OF  COMPUTERIZED  DATA  BASES  ON  A  TOPIC 
RELATED  TO  YOUR  RESEARCH  OR  A  PROBLEM  YOU  ARE  WORKING  ON? 


We  are  looking  for  questions  to  search  In  connection  with  an  NSF  sponsored 
project  Investigating  the  processes  In  online  searching  and  the  performance  of 
searchers. 

We  will  conduct  a  very  comprehensive  searchbf  your  question  and  provide 
you  with  the  answers  (In  the  form  of  abstracts)  FREE  OF  CHARGE. 

As  an  Information  user  we  will  ask  you  to: 

1.  Provide  your  question  In  a  written  form. 

2.  Participate  In  a  short  Interview  about  the  nature  of  the 
problem  you  are  working  on. 

3.  Fill  but  a  very  short  questionnaire . 


4.     Indicate  which  answers  were  relevant  to  your  question  and  if 
the  search  was  satisfactory. 

Your  participation  will  provide  you  with  a  comprehensive  free  search.    It  will 
provide  us  with  crucial  data  derived  from  real  information  needs  and  questions. 


We  hope  that  you  will  agree  to  participate  in  this  research  effort  I 
Please  contact  by  phone  or  fill  out  the  form  below: 

Tefko  Saracevic,  Principal  Investigator 

368-3610  or 
Alice  Chamis,        Project  Manager 

368-3501  YOU 


TO:    Alice  Chamis,  Ph.D 

Matthew  A^  Baxter  School  of  information  and  Library  Science,  CWRU 

-Baker  Building 
Cleveland i  Ohio  4A1D6 

I  am  willing  to  provide  a  request  for  searching! 


Name ;   Phonet  

Department  and  address: 


When  is  the  best  time  to  call  you: 

Matthew  A.  Baxier  School  of 
^  ^ormation  and  hibroTy  Science 

ERIC 


Form  2 


EXPERIMENTS  IN  INFORHATION  SEEKING  AND  RETRIEVING 
SEARCHER  PROFILE 

Please  complete  the  foiibwlng  questlpxmalte  lo  that  ire  can  plan  these 
escperlments  to  use  your  expertise  and  to  schedule  you  for  online 
searching  at  your  convenience. 

Name  —  —   

Address    


Telephone  B    H 

When  is  the  best  time  to  call  you?   

How  frequent iy  do  you  search  Dialog?   

What  is  your  subject  expertise?   

What  databases  do  you  search  most  often? 


What  thesauri  do  you  have  available? 


What  type  of  terminal  do  you  use  for  searching? 

Couyuter  terminal    

Microcomputer   

Number,  by  order  of  preference,  the  days  you  can  search «  Please 
indicate  a.m.  or  p.m. 

 Hon.    -  tues.    -     -     Wed,   Thurs.   Fri.   Sat. 

Comments : 

If  you  have  auy  questions,  call  Alice  Chamis,  Ph.D.  at  Case  Western 
Reserve  University,  368-3501. 


485 


Form  2a 


EXPERIMENTS  IN  INFORMATION  SEEKING  AND  RETRIEVING 
SEARCHER  PROFILE 

Please  cbiiQ>iete  the  fdlldwlzig  questionnaire  so  that  we  can  plan  these 
experiments  to  use  your  expertise  and  to  schedule  you  for  online 
searching  at  your  convenience* 

Name   

Address  


telephone  B   H 

When  Is  the  best  time  to  call  you?  

How  frequently  do  you  search  Dialog?   

What  Is  your  subject  expertise?  -   

What  databases  do  you  search  most  often? 


What  thesauri  do  you  have  available? 


What  type  of  terminal  do  you  use  for  searching? 

Computer  terminal  —   . _   

Microcomputer  —   

Number,  by  order  of  preference,  the  days  you  can  search*  Please 
Indicate  a«m«  or  p.m* 

 Moni   tues*   Wed*   thurs*   Frti   Sat* 

introductory  meeting  wiii  be  held  Saturday,  September  28,  1985  from 
9:30  until  noon.  Hatch  Auditorium,  1st  floor  Baker  Building, 
Case  Western  Reserve  University* 

i  can  attend  — 

i  cannot  attend           -       Alternate  date:   

Cogent 8 :  (use  back  of  sheet) 

If  you  have  any  questions,  call  Alice  Chamis,  Pb«D*  at  Case  Western 
Reserve  University,  368-3501* 


For  Project  Pse  Ottly 

Searcher  Number  :  :  : 

Question  Number       :    ;  ; 

Form  3  (hnmi)' 


NSF  PROJECT 

EXPERIHENTS  ON  COGNITIVE  ASPECTS  OF  

INFORMATION  SEEKING  AND  INFORMATION  RETRIEVING 

Searcher  Quest ioxmalre 


Soc  Sec  No« 


Last  Name  First  Name 


Street  Address 


City  State  Zip 


Business  phone:  ;;;;;;;;;;  Home  phone: 
1.    When  can  we  best  call  you?   


2.    How  often  do  you  search  DIALOG?  (Please  circle  the  best  estimate.) 

5    Daily         4    Twice  a  week         3    Once  a  week         2    Twice  a  month         1  Less 

3*    Refer  to  "List  A  -  Selected  Databases'*  (attached)  and  please  indicate  those  databases 
that  you  search  most  ofteni  in  order  of  decreasing  use.    Below  each  database  code  indicate 
how  often  you  search  it»  using  the  same  codes  as  in  Question  2  above. 

Databases  used  most  often 

^f\T%XP  ••••  •••  ••• 

s^yjust  ••••  •••  •••  ••• 


4.    Frequency  of  use  of  the  above  databases. 


5.  Ifpw  please  refer  to  ''List  B  -  ThesAuri"  (attached)  and  indicate  those  most  important  to 
you  when  you  search. 

CODE*   -Ts — Z-  — -t- —  s     - T-^  — t — X — -t--   — T;  __• — 5 — *  —  t--        z      T* —  ■  —  ■_         T  • —  •     •  • 

6.  What  are  your  preferred  times  for  working  on  this  project?    Indicate  order  of 
preference,  using  codes  listed  below: 

First,       2nd,  3rd, 


SUN 

HON 

tUE 

WED 

THU 

FRI 

SAT 

f:dd-i2:ee 

Al 

A2 

A3 

A4 

A5 

AS 

A7 

PM: 

1:30-  4:3d 

Pi 

P2 

P3 

P4 

P5 

P6 

P7 

EVE: 

5:30-  8:30 

El 

E2 

E3 

E4 

E5 

E6 

E7 
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LIST  A  -  SELECTED  DIALOG  DATABASES 


AGRieULtURE  AKD  NUTRITION 


010 
005 
051 


AGRleOLA 
BIOSIS 

FOOD  SCIENCE  AND  TECHNOLOGY  ABSTRACTS 


BUSINESS/ECONOHiCS 

557         MOODY'S  CORPORATE  NEWS— INTERNATIONM. 
556         MOODY'S  CORPORATE  NEWS~U.S. 
132         STANDARD  &  POOR'S  NEWS 


INDUSTRY  SPECIFIC 

168         INSURANCE  ABSTRACTS 

042         PHARMACEUTICAL  NEWS  INDEX 


MARKET  RESEARCH;  INDUSTRY;  MANAGEMENT 

015  ABI/INFORM 

139  ECONOMIC  LITERATURE  INDEX 

122  HARVARD  BUSINESS  REVIEW 

075  MANAGEMENT  CONTENTS 

018  PTS  F  &  S  INDEXES 

016  PTS  PROMPT 

148  TRADE  &  INDUSTRY  INDEX 


STATISTICAL/DEMOGRAPHIC  DATA 
575         DONNELLY  DEMOGRAPHICS 


CHEMISTRY 

138         CHEMICAL  EXPOSURE 

174  CHEMICAL  REGULATIONS  AND  GUIDELINES  SYSTEM 
240  PAPERCHEM 


COMPUTER  SCIENCE 

256  BUSINESS  SOFTWARE  DATABASE 

008  CroiPENDEX  

275  C^UTER  DATABASE 

013  INSPEC  _ 

232  .MENU~THE  INTERNATIONAL  SOFTWARE  DATABASE 

233  MICROCOMPUTER  INDEX 
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Form  3-  page  3 


LIST  A  -  SELECTED  DIALOG  DATABASES  (Contlaued) 


CURRENT  AFFAIRS 


259 

AF  NEWS 

647 

MAGAZINE  INDEX 

lit 

NATIONiO.  NEWSPAPER  INDEX 

049 

PAIS  IN^RNAtlONAL 

132 

STANDitif(D  AND  POOR'S  NEWS 

260 

UPI  NEWS 

184 

WASHINGTON  POST  INDEX 

167 

WORLD  Affairs  import 

EDUCATION 
001  ESiC 

054         EXCEPTIONAL  CHILD  EDUCATION  RESOURCES 


ENERGY  And  ENVIRONMENT 

112  AQUACULTURE 

005  BlOSiS 

069  ENERGYLINE 

040  ENVIRONLINE 

041  POLLUTION  Abstracts 


LAW  AND  GOVERTOffiNT 

102  ASi 

101  CIS 

174  CHEMICAL  nBGULATlONS  MD  GUIDELINES 

135  CONGRESSIONAL  RECORD  ABSTRACTS 
171  CRIMINAL  JUSTICE  PERIODICAL  INDEX 

020  FEDERAL  INDEX 

136  FEDEtb^i  REGISTER  ABSTRACTS 
244  LABORL^ 

150  LEGAL  RESOURCES  INDEX 

021  NCJRS 
006  NtiS 


MATERIALS  SCIENCE 

032  METADEX 

118  NONFERROUS  METALS  ABSTRACTS 
240  PAPERCHEH 

033  WORLD  ALUMINUM  ABSTRACTS 
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Form  3  -  page  4 


LIST  A  -  SELECTED  DIALOG  DATABASES  (Continued) 


MEDICINE 


0Q5 

BIOSIS 

1  Jo 

07  A 

TMTVDMATTHMitT     1>II  ADMAOVf  TTT/^  AT  AOCTDA/^TC 

IMl&KIiAXlUrlAij  IrnAKnAUfiUXlCfAL  ADdXKAUlo 

yfy 

MVT^T  TU17 

UoD 

01  Q 

Zio 

MTTDCTWP    A1ffT\    AT  T  T17f\    III?  AT  1*11    f  mJltAtlT  \ 

161 

OCCUPATIOKAL  SAFETY  AND  HEALTH  (NIOSH) 

042 

PHARMACEUTICAL  NEWS  INDEX 

23  o 

185 

ZOOLOGICAL  RECORD 

1  AND  TcUHNOLlAvX 

AAO 

COMFENDEX 

U07 

A1  Q 

INSFEC 

m  A 
Ul<f 

TG117P 

OOp 

NTIS 

1  1  o 

WORLD  TEXTILES 

SOCIAL 

SCIENCES  AND  HUMANITIES 

038 

AMERICA:  HISTORY  AND  LIFE 

056 

ARTBIBLIOGRAPHIES  MODERN 

064 

CHILD  ABUSE  AND  NEGLECT 

291 

FAMILY  RESOURCES 

03t 

HISTORICAL  ABSTRACTS 

036 

LANGUAGE  AND  LANGUAGE  BEHAVIOR  ABSTRACTS 

071 

MLA  BIBLIOGRAPHY 

049 

PAIS  INTERNATIONAL 

Oil 

PsychlNFO 

097 

RILM  ABSTRACTS 

037 

SOCIOLOGICAL  ABSTRACTS 

167 

WORLD  AFFAIRS  REPORT 

ERIC 
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Form  3  -  page  S 


LIST  B  -  THESAURI 


AGRICULTURE  AND  NUTRITION 

TOlO       Agricultural  Terms 
T005       BIOSIS  Search  Guide 
T051       FSTA  Thesaurus 


INDUSTRY  SPECIFIC 

T168       Thesaurus  (INSURANCE  ABSTRACTS) 


MARKETING  RESEARCH;  INDUSTRY;  MANAGEMENT 

T015       Siarch  IKIFORM 

T122       HBR/Onlltue  Thefautus 

T075       MANAGEMENT  CONTENTS  Database  Thesaurus  and  Indexers  Manual  and 

Dictionary 
T018       PTS  Users  Manual 
T148       Subject  Guide  to  lAC  Databases 


CHEMISTRY 

T174       CRGS  Thesaurus 

T240       Keyword  Frequency  List  (PAPERCHEM) 


CGHLPUTSR  SCIENCE 

T256       Search  SOFTWARE. 

T008       SHE:  Subject  Headings  for  Engineering 

T275       The  Compul:er  Database  Thesaurus  and  Dictionary 

T013       INSPEC  Thesaurus 

T233       MicroTlps»  A  User'^s  Guide  to  Microcomputer  Index  on  DIALOG 


EDUCATION 


TOOl       Thesaurus  of  ERIC  Descriptors 


ENERGY  AND  ENVIRONMENT 

Til 2       Aguacuiture  Thesaurus 

T005       BIOSIS  Search  Guide 

T041        POLLUTION  ABSTRACTS  Thesaurus 


EKLC 
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Form  3  ^  p^ge  6 


LIST  B  -  THESAURI  (Continued) 


LAW  AND  GOVERNMENT 

tIOI  CIS  Online  Users  Guide  and  Thesaurus  of  Index  Terms 
T174       CRGS  Thesaurus 

T171        Thesaurus  (CRIMINAL  JUSTICE  PERIODICAL  INDEX) 
Tb21       National  Criminal  Justice  Thesaurus 
Tbb6       COSATI  Subject  Category  List 


MATERIALS  SCIENCE 


T032       Thesaurus  of  Metallurgical  Terms 

Thesaurus  of  Pulp  and  Paper  Terms 
TOSS       Thesaurus  of  Aluminum  Technology 


MEDICINE 


TpOS  BIOSIS  Search  Guide 

Tb74  Thesaurus  of  Subject  Terms  (IPA^ 

TC76  Theaaurus— LIFE  SCIENCES  COLLECTION 

T154  Medical  Subject  Headings 

Tb86  MENTAL  HEALTH  ABSTRACTS  Users  &ilde 

T218  CINAHL  Subject  Headings 

T238  Teleglnllne  User's  Manual 


SCIENCE  AND  TECHNOLOGY 


T008  SHE:  Subject  Headings  for  Engineering 

Tb89  GeoRef  Thesaurus  and  Guide  to  Indexing 

Tbl3  INSPEC  Thesaurus 

Tbl4  ISMEC  Thesaurus 

Tbb6  COSATI  Subject  Category  List 

T119  Textile  Technology  Dlge8t~Keyterms 

Tb67  Register  of  keyterms  (WORLD  TEXTILES) 


SOCIAL  SCIENCE  AND  HUMANITIES 

Tb64  Child  Abuse  and  Neglect  Thesaurus 

T036  LLBA  User's  Manual 

Td49  PAIS  Subject  Headings  List 

fbli  Thesaurus  of  Psychological  Index  Tetms 

Tb97  RILM  English-Language  thesaurus 

Tb37  SOCIOLOGICAL  ABSTRACTS  User's  Manual 


EKLC 
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VBF  HOJBCT  iST-850  5411 
EXPERIMENTS  ON  IHE  OOGNITiVE  ftSI^CTS  OF 
INF^MATION  SEEKING  AND  INFOI^TION  ^SSSIEVWG 


NSF  Project  Surtina^y 

Biis  project  will  study  the  effects  of  a^a^iety  of  factors  ch  the 
effectiv«iess  and  efficiency  of  online_ Marches  perfcw^  of  questions. 

The  objective  of  this  project  is  to  conduct^  series  of  experiments  on  the  A)  context 
of  requests  in  informatiori  retrieval   B)  structure  and  classification  of  requests 
C)  procedures  in  request  analysis  and  searcdi  strategyr  ahdr    D)  cJiaracteristics  and 
behavior  of  intentiediary  searchera. 


IT^TROCTi 


lhahk  you  for  e3q>ressing  an  interest  in  stfatdttfaig  a  question  to  this  NSF  Project, 
%mic^  will  provide  you  %rf.th  a  free  online  database  search  for  ydiir  question. 

1.  Please  coiplete  the  form  below  md  tine  attadied  "Question  Request  Porm"  so  that  we 
can  conduct  your  literature  seardi  and  scfedule  you  for  a  follow-^p  interview. 


Last  Name  First  Name 


Street 


City  State  Zip 

T^l^hcne 


l^siness  Hone 
CWRU  Campus  address; 


Department  Building  Room 

Canpus  Ebtt.    — : — t-  t — t- 

2.    When  is  the  best  time  to  call  you?  


EKLC 
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3.    Please  indicate  belopw  a  specific  time  or  tiines  daring  tire  y^k  of 

  (  9^,m,  -  4:30  pjn>) 

you  ocxild  be  available  for  an  interview, 

(Wb  will  call  you  to  ocnfirm  a  3^)ecific  time  and  place.) 


4.    The  NSF  Praject  Off  ice  is  located  Jft  Buildingr  Rocm  3pr  eW!^ 

Biterviews  can  be  held  there  or  at  your  office.    Please  indicate  your 


Bakerr  Rcxm  318  Your  officer  Bldg.    !tocm_ 


5.    The  interview  concerning  ywjr  quest io^will  be  taped^fw^t^  use  of  the 
project  only.    Please  sign  below  to  confirm  that  you  agree  to  teving  this 
search  interview  taped  and  that  you  will  provide  a  determination  of  the 
relevance  of  each  of  the  attracts  r  obtained  from  the  seardir  within  two 
weeks  of  the  receipt  of  the  abstracts. 

Signature     -   Date    


The  Project  Off  ice  reserves  the  right  to  maintain  c^ies  of  all  searches. 
All  information  bbtained  for  this  project^ili  be  treated  confidentially  and 
all  copies  maintained  will  be  held  confidentially  in  tenr^  of  the  person  v*io 
requested  the  search.    If  you  have  further  questions r  please  call  us  at 
216/368-3501. 

Mice      ehamiSy  SSiDi 
Inject  Manager 
Tefko  SaraceviCr  Hi.D. 
i^incipal  fiivestigator 


Form  4  (10/1 '8 S)    .  p^ge  2 
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For  Project  itee  CftJy 
Question  No.    Qt  - 
Searchers     :    4  —t-  — 
-  z-   t  ^  - 

Database    -  t    %  ^  - 

Ptoriti  5  (10/1/85) 


 M5F  racgBCT_IST--85d  5411 

EXPEaRlMENTS  ON  TOE  COGNITIVE  ffiPBCTS  INPCWttTION 
SEEKING  AND  INPOIWATldN  RETRIEVING 


QUESTION  REQUEST  FOm 


R.    Please  state  ycpr  question  belo«7  as  clearly  and  gpecif ical^^s  

possible,    "niis  statenient  of  your  question  will  be  the  only  information  made 
available  to  most  of  the  searchers  of  the  online  datateses  j«ho  w^ 
your  literature  search.    Define  any  %«ords  v^ich  have  special  neanihg  for 
your  <^estiQn^ 

1.  Brief  title:   -      

2.  Question  statement: 


(If  necessary,  use  the  back  of  the  page.) 


4  m 


Fbr  eacii  question  below  circle  the  nunber  %*ich  oorrespcnds  to  a  description 
of  the  t^pe  of  seairch  ycxi  want. 

B.  bo  ycHi  want  a  precise  cx:  a  broad  search? 

3  A  precise  seardi  produces  a  relatively  annall  set  of  abstracts 
eadi  of  whidi  is  likely  to  be  relevant. 

4  A  broad  seardh  produces  a  relatively  larg^  set  of  abstracts  with 
a  better  diance  of  containing  all  relevant  abstracts  in  the 
database. 

C.  TVpe  of  a^iicatidn  of  this  researdi: 

5  tSidergraduate  study        8  Ihdustrial 

6  a^aduate  stud^  5  General 

7  Faculty  research  10  Other     

(Please  specif) 

p.  Do  yod  want  to  place  restrictions  on  the  language  of  publication  of  the 
articles  retrieved? 

11  Biglish  only 

12  Any  language 

Ei    Do  you  want  to  restrict  the  years  of  publication  of  the  articles 
retrieved? 

13  East  5  years 

14  m  limits 

15  Otherr  specif   19  to  19  

P.    If  you  are  faniliar  wiUi  the  DIHDG  databases  r  please  indicate  those 
that  would  be  z^ropriate  for  your  question. 

16   

17   -  - 

18  


Form  5  {l6a/8S)    -page  2 


REr  NSF  Project  ••Experiments  on  the  Cognitive  Aspects  of   Information  Seeking 
and  information  Retrieving*' 


We  have  received  your  question  and  would   like  to  interview  you  furthei 
about  your  research.     The  interview  itsell   wil).  not  take  longer  than  a  half 
hour.     As  per  the  times  you  preferred ,  we  would  like  to  schedule  your  intervii 
for:        -   _   .  


Please  call  us  at  368-3501  any  time  between  9  a.m.  and  5  p.m., 
Hon.  through  Fri,  if  you  cannot  come  at  this  time.  Ule  will  be  glad  to 
reschedule  ybu  for  a  time  that  is  convenient. 

We  appreciate  your  participation! 

Donna  Trivison 
Research  Assistant 
Alice  y .   Chamis,   Ph • D. 
Project  Manager 
Tefko  SaraceviCu  Ph.D^ 
Principal  investigator 


Form  6  <10/14/85) 
f ii«stest 


In'terviewi                                                                            For  Project  Use  Only 
ba1:e/time   .   Question  Noi  :  s  : 

Loca£ion   Project  Searcher  No.  s_  s  

Form  7  (10/14/85) 

1 n t e rviewer  

iNTERViEW  QBEStlONNAiRE 

Purpose: 

Your  question  will   be  searched  by  several  searchers  based  on  the  written 
question  which  you  have  submitted.     The  purpose  of  this  interview  is  to  tell  i 
about  the  problem  that  motivated  your  question.     The  information  will  be 
used  to  design  an  additional   search  for  your  question^ 


1.     Could  you  describe,    in  about  5  minutes,   the  problem  which  motivated  your 
research  question . 

Zm     In  your  opinion  and  on  a  scale  of   from  1  to  5,   would  you  describe  your 
problem  as  weakly  defined  or  clearly  defined,   with  1   being  weakly  defined  and 
being  clearly  defined. 


1  2  3  4  5 

Meakly  Defined  Clearly  Defined 

PROBLEM  DEFINITION  SCALE 


3.     How  do  ybu  expect  to.  use  the  information  obtained?  _<e.g., 
diBsertation/thesis,   project  research,   writing  a  proposal ,  administrative 
jblannihg,   professional   awareness,   review  article/book^   curriculum  development 
classroom  material ) 


4.     On  a  scale  of   from  1   to  5^   would  you  say  that  ycDur  use  of  

this  information  will   be  open  to  many  avenues^   or,   for  a  specifically  defined 
purpose,   with  1   representing  open  to  many  avenues  and  5  representing  a 
specifically  defined  purpose. 


•  2 ________ •  -  ---- -  8  _____________ : 2 

Open  to  Many  Purpose  is  Narrowly 

Avenues  _   _     _    _        _  Defined 

INTENT  SCALE 


For  Project  Use  Only 

Ques'tion  No.   s  s 

Project  Searcher  l^o.   s  s 

Form  8  C10/1B/8B> 


CONTEXT  QUESTiONNAIRE 
<USER> 


1.     On  a  &cale  of  1  to  S ,   how  would  you  rank  "the  probability  that  information 
about  the  problem  which  motivated  thjis  research  questxon  may  be  found  in  the 
literature. 


Highly  Improbable  Highly  Probable 

That  It  Exists  That  it  Exists 

<       >     i  don't  know. 

PROBLEM-PUBLie  KNOWLEDGE  SCALE 


2.  On  a  scale  of  1  to  5,  how  would  you  rank  the  amount  of  knowledge  you 
possess  in  relation  to  the  problem  which  motivated  this  request. 


12  3  4  5 

tittle  PerBonal  Considerable  Personal 

knowledge  Knowledge 

INTERNAL  KNOWLEDGE  SCALE 
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i 


ate/Time^..  '         QueB'bion_Nb.   ,  8  

Location  ^  Project  Searcher  No, 

Interviewer   Form  9  Cll/13/S5> 


INTERVIEW  EVALUATIONAND  CONTEXT  QUESTIONNAIRE 
(PROJECT  SEARCHER) 

1 iH  your  opinion  and  on  a  ecale  oi   irom  1  to  5^  would  you  desaribe  the  user 
problem,  as  discernible  from  the  interview,  as  weakly  defined  or  clearly 
defined,  wi^ch  1  being  weakly  deixned  and  5  being  clearly  defined. 

I  8   s  s  :  *   : 

~     "l""/"  "  2  3  4   5 

Weakly  Defined  Clearly  Defined 

PROBLEM  DEFINITION  SCALE 


2;     On  a  scale  of  from  1  tQ  5,  would  you  say  that  the  use  of  this  information 

this  user  will  be  open  to  many  avenues,  or,   for  a  s»pecif  ical  ly  defined 
purpose,  with  i  representing  open  to  many  avenues  and  5  representing  a 
specifically  defined  purposes 

S  m  S  ^  5  5  _        ^  5 

 2^"     ~  3""  4  '  B   

Open  to  Many  Purpose  is  Narrowly 

Avenues  Defined 

INTENT  SCALE 


3i     On  a  scale  of   i  to  5^   how  would  you^  as  a  project  searcher,   rank  the 
probability  that  information  about  the  problem  which  motivated  this  research 
question  may  be  found  m  the  literature. 


1   2  3  4  5  

High ly  improbable  Highly  Probabii 

That  It  Exists  that  it  Exists 

i       >     i  don't  know. 

PRGBLEH-- PUBLIC  KNOWLEDGE  SCALE 


4.     On  a  scale  of   1  to  &^   how  would  you^   as  a  project  searcher^   rank  the 

amount  of  knowledge  you  possess  in  relation  to  the  problem  which  motivated 
this  request. 

s  :  _   :  _s  :  s 

"  2              ^         3                        4  "  5 

Little  Personal  Considerable  Personal 

Knowledge  _      _  Knowledge 

INTERNAL  KNOWLEDGE  SCALE 
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For  Prbject  U&e  Only 


Ques'ti.bn  No. 


Project  Searcher  No.   s  s^^s 

Form  10  (10/15/85) 


CONTEXT  QUESTIONNAIRE 
(SEARCHER) 


1.     On  ~sl  scale  of  1  -to  5,  how  would  you  rank  the  probability  that  information 
about  the  problem  which  motivated  this  research  question  may  be  found  in  the 
literature. 


Highly  improbable  Highly  iProbabic 

that  It  Exists  That  It  Exists 

<       >     i  don't  know. 

PROBLEH-PdBbiC  KNOWLEDGE  SCALE 


2i  On  a  scale  oi  1  to  &^  how  would  you  rank  the  amount  at  knowledge  you 
possess  in  relation  to  the  problem  which  motivated  this  request i 


12  3  4  5 

Little  Personal  Considerable  Personal 

know ledge  Know 1 edg  e 

INTERNAL  KNOWLEDGE  SCALE 
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I~  wi>  III      *  *  '%  w  #^  ##ww^*A*w* 


QliEStidN  NUMBER  

ANALYSIS  FOR  STRUCTURE 


7LEA&-1N) <«ODIFIERS) (QUERIES) CMODIFIERS) (SUBJECTS) (CONSTRAINTS) 


bcmain:     Number  of   (Dial index)  Categories 

Clarityi     1.     Semantics  (meaning  of  terms) 

C  3E  3E  □£  3 

1  2  3  4  5 


2.     Syntax   (relation  between  terms) 


□  E  □  E 

1  2 

Clarity  Score: 


Specificity: 
query  Is  E 
meta 

query  2:  E 
meta 

query  3:  E 
meta 


subject  1 :  E 
meta 

subject  2:  E 
meta 

subject  3:  E 
meta 


□  E 


□  E 


□  E 


□  E 


□  E 


□  E 


3  E 

3  E 

3  C 

3  E 

3  E 

3  E 


3  E 
3  A 


i. 


3  E 
3  4 

3  E 
3  4 

3  E 

3  4 


3  E  3 
5 


/  2  = 


2i 


3  E  3 

5  object 

3  E  3  

S  object 

3  E  □  

5  object 


Query  Mean 


3  E  3  E  3 

3  4  5  object 

3  E  3  E  3 

3  4  5  object 

3  E  3  E  3 

3  4  5 

Subject  Mean   


Specificity  Score:  ^  +  .  _  /  2  = 

Query  Mean         Subject  Mean 

Complexity:       Number  of  Search  Concepts 

Number  of  Constraints 

Presuppositions:  Number/ Percent  transferable  to  search  process 

Final  Summary  Soore: 
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User  Name  _  _  ^  Question  No. 


Oa£e   Number  of  Abs'trac'tB  

Form  12  Cl/2/86>   £ i le: abstracts 


USER  QUESTIONNAIRE  ON  EVALUATION  OF  ANSWERS 


!•     How  much  time  did  you  spend  reviewing  these  abstracts? 

2m     In  ah  overall   sehse^   ii   you  were  a&ked  to  assign  a  dollar  value  to  the 
usefulness  of  this  entire  s€?t  of   abstracts  to  ybu,   what  would  that  dollar  vali 
be?  *  i  cannot  assign  a  dollar  value. 

3.  Could  ybu  rate  your  participation  in  this  project  and  the  information  tha* 
has  resulted  from  it  ass 

5  Worth  much  more  than  the  time  it  has  taRen 

4  Worth  somewhat  more  than  the  time  it  has  taiken 

3  Worth  about  as  much  as  the  time  it  has  taScen 

2  Worth  less  than  th<5?  time  it  has  taken 

1  Practically  worthless 

4.  PROBLEM  RESOLUTION  SCALE  -  Oh  a  scale  of  1  to  5,  what  contribution  has 
this  information  made  to  the  resolution  of  the  problem  that  motivated  your 
question? 

•  •   m  .  .    ....  .    J     • 

^  2  3  4  5 

nothing  contributed  substantial  contribution 


5.  SATISFACTION  SCALE  -  On  a  scale  of  1  to  5 ,  how  satisfied  were  you  with  th« 
results  of  the  search? 


1  2  3  A  5 

dlssatisf  ied  satisf led 

&m  bo  you  have  any  general  comments  about  any  part  of  the  project,  including 
the  questionnaires,  the  interview^  or  the  results?  (Use  the  back  of  the  page 
If  necessary 


ERIC 
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Ques-kion  Noi        :  :  : 

Search  Ordfcr     1     2     3     4     5  6 

Searcher  Noi   :  s  : 

Form  13  til/4/85S 
file:  Bearchrecord 

SEARCH  RECORD  FORM 

Database  to  be  Bi?arched: 

Thesauri  used: 

Preliminary  search  strategies: 


EKLC 


 NSF  Projebt  IST-^S5D5^11 

*'EKperiment&  in  the  Cognitive  Aepects  oif  Information 
Seeking  and  Information  Retrieving'* 

User  Name  Quest i on  No. 

Date      .  _  Abstracts  Enclosed 


Firct  of   al 1 ,   thank  ybu  for  your  willingness  to  participate  in  the 
experiments  dealing  with  online  searching. 

Your  Question  wa^  searched  by  several   searchers  in  the  database , 


The  enclosed  set.  of   abstracts  repres£>ntri  tt^e  combined  answers  retc^ieved  by 
all  searchers. 

Pe  havs^  prcvided  yau  with  a  duplicate  set  of  abstracts.  Keep  the 
duplicate  for  your  use.  Please  indicate  your  r£?levahce  ,5udgements  on  the 
originals  and  return  them  to  the  Project  Office  in  the  enclosed  envelope. 

Each  abstract  should  be  evaluated  according  to  the  degree  of  its 
relevance  to  the  qu -^stibn  you  submitted  for  searchihsg.     A  copy  of  your  questic 
is  attached.     The  degree  of   relevance  is  to  be  determined  using  the  following 
three  point  scale: 


RELEVANT  -  Any  document  which  on  the  basis  of  the  inf brmatioh  it  conveys  is 
considered  to  be  related  to  ycur  question ,   even  if   the  information  is 
outdated  or  already  familiar  to  you. 

PARTIALLY  RELEVANT  -  Any  document  which  on  the  basis  of   the  information  it 
conveys  is  considered  only  somewhat  or  in  some  part  related  to  your  question  c 
to  any  part  of   your  question. 

NONRELEVANT  -  Any  document  which  on  the  basis  of  the  information  it  conveys  is 
not  at  ail   related  tb  your  question. 

After  you  have  evaluated  each  document^   please  complete  the  enclosed 
"User  Questionnaire  on  Evaluation  of   Answers,"     Return  it  with  the  abstracts. 

your  evaluation  will   provide  the  data  necessary  for  the  completion  of 
our  study  of   the  factors  which  affect  online  searching. 

Please  call   our  office^  ,    if   you  have  any  questions  about  the 

evaluation.     Your  cooperation,   as  m  the  past,   is  greatly  appreciated. 


Sincerely , 


Al ice  y .  Chamis ,  Ph . D. 
NSF  Project  Di  rector 


Return  to:     Alice  Y.   Chamis,   Ph;D.  Please  return  by   

Baker  Building ,   Room  318 

Case  Western  Reserve  University 

Cleveland,   OH  44106 

  F'ormi  4  <  i  /2/86 >  f  i  1  e :  answers 


Enolosares 


Form  IS  (3/31/S6) 


  _  NSF  PROJECT  1ST-S50  5411 

EXPERIMENTS  ON  THE  COSNITIVE  ASPECTS  OF  INFORMATiON 
SEEKING  AND  INFORMATION  RETRIEVING 


USER  QUESTIONS  FOR  ONLINE  SEARCHES 


QUESTION  NUMBER: 
DIALOG  DATABASE  USED: 
SEARCHER  CODE  NUMBERS: 


Al.  BRIEF  TITLE: 


A2.   QUESTION  STATEMENT: 


TYPE  OF  SEARCH  REQUESTED 


Bi  A  precise  or  a  broad  search: 

Ci  Research  appi ica'bion: 

Di  Retrieve  articles  in  English  only  or  any  language; 

Ei  Years  to  be  searched:  

Fi  DIALOG  databases  suggested: 


SEARCH  TERM  OVERLAP 


NAME   TODAV'S  DATE 

TIME  SPENT 

Question  Number  --  --  --  --  -- 


Searcher  Search  Term  Overlap  wi-bh  Searcher  (I-bemized? 

Number  • 


Form  16 

<4/21/S6) file: overlap 
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APPENDIX       PROCEDURES  USED  FOR  THE  PROJECT  (NUMBERED  1  THROUGH  12) 

Procedure 
Number  Title 


1  Procedures  for  Receiving  the  Written  Question  Statements 

2  Guide  for  Interview  of  User  by  Searcher 

3  Searcher  Instructions 

4  Procedures  for  DIALOG  Searching 

5  Procedures  for  Recording  Searches  from  Hard  Disk  onto  Diskette 

6  Procedure  for  DIALOG  Searching  Using  the  Compaq 

7  Procedures  for  Loading  the  Compaq  Searches  from  Diskette  onto 
IBM/XT  Hard  Drive 

8  Procedures  for  Project  Searches 

9  Procedure  for  Determining  Search  Term  Overlap 

10  Procedures  for  Completing  Question  Request  Forms  Using  the 
IBM/XT  Computer 

11  Procedures  for  Using  the  Dbase  Fllies  on  the  Compaq  Computer 

12  Procedures  for  Using  Pascal  Programs  -  Data  Processing  Report 
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Procedure  No.  1 
C10/16/8B:) 


Procedures  for  Receiving  UJri't'ten  Ques'tion  S'ta-bemenbs 


1.  AssiQh  ques'tibn  a  number  from  "the  mas-Lei-  list,  of  ques'tions  received.  Uri 
qu&s'tion  number  on  each  page  oi   ques'tibh  form. 

2.  Set  up  a  folder  for  each  que&'tion  using  "the  ques-tion  number  aB  -the  'ti'tle 
-the  folder. 

3.  hake  copies  for:     da-ta  processing   Kl   copy;)   -  give  -to  v)eong 


5.  Prepare  interview  forms  listing  date/ time  and  question  number.  Put  these 
forms  in  "'^he  que&tioh  folder. 

4.     Assign  quesition  to  searcher. 


interview   (l  copy) 

searchers   (5  copies  ^   pp.   3  &  4  ovily*        put  all  of  these 

copies  m  folder 


Set  up  interview: 


a  m  ass  j.gn  to  interviewer 

b •  decide  on  time  and  place 

c.  notify  user 

d.  record  on  calendar 
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Procedure  No,  2 

1 10/1 4/ 85:)   1 1  le:  interview 


GLiiDE  FOR  INTERVIEW  OF  USER  BY  SEARCHER 


1.  Contact,  the  u&er  at  least  one  week  ahead  of  time  to  schedule  an  interview 
Determine  the  date,  time  and  place  ior  the  interview. 

2.  Contact  the  user  the  day  before  the  Interview  to  confirm  the  scheduled 
interview. 

3.  Before  going  to  the  interview,  record  on  the  tape  recorder  the  question 
number «   your  searcher  number,   the  date  and  time  of  the  scheduled  interview. 

4.  Take  to  interviews  1 ^  copy  of  the  user's  question  request  forms  (all  4 
pages:!     2>  an  "Interview  Questionnaire,"  and  3)  a  '•Context  Questionnaire", 


5.  Start  the  interivew  in  a  friendly  manner  and  describe  the  procedures  for 
the  interview,  the  taping,  the  search  and  evaluation  of  the  results. 

6.  Explain  that  the  purpose  of  the  interview  is  to  obtain  additional 
information  about  the  question  for  additional  searches  of  the  question. 


7.  Ask  the  user  the  questions  noted  on  the  "interview  Questionnaire."  Do  no 
use  any  additional   prompt  beyond  the  question. 

S.     €ive  user  the  written   "Context  Questionnaire"  to  fill  out. 

8.  Thank  the  user  for  his  or  her  time  and  inform  him  or  her  that  he  or  she 
will   receive  the  abstracts  for  relevance  evaluation  within  three  weeks.  One 
set  of   the  abstracts  should  be  marked  for  relevance,   using  the  scale  provided 
and  returned  to  the  Project  Office  within  2  weeks.     A  second  copy  of  the 
abstracts  are  for  the  user  to  keep. 


Alice  y.  Chamis,  Ph.D. 
Project  Manager 
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PX'DcedureB  No .  3 
Cll/4/85:) 

file:  ins'truc't 


NSF  Project 
SEARCHER  INS  T  RUC  T IONS 
<  Prel iminaries) 


1.  VdU  have  be>en  aBBiqned  B  or  6  questions  to  search  on  Dialog  using  the 
databases  available  on  the  Classroom  Instruction  Prociram. 

2.  Read  each  question  as  described  by  the  user  on  his  or  her  Question  Requei 
Form.  Each  question  should  be  searched  in  the  order  and  database  specified  < 
the  Search  Record  Form  supplied  to  you  for  that  question. 

3.  After  readihq  all  of  the  assigned  questionsv  examine  the  thesauri  availal 
in  the  NSF  Project  Office  to  determine  the  isearch  vocabulary  needed.  Note  oi 
the  Search  Record  Forms  the  thesaUri  used. 

4.  Before  going  online,   formulate  the  search  strategies  lor  each  question 
which  will   provide  the  best  retrieval.     Record  these  preliminary  search 
strategies  br»  the  Search  Record  Forms.     Use  as  many  forms  as  needed. 

5.  Underline   Cor  highlight.)   the  terms  you  don't  understand  in  each  question 

6.  Vbu  should  now  be  ready  to  conduct  your  searches  online.  Instructions  f< 
online  searching  are  written  on  the  next  page. 

DIALOG  TIPS 

1.      fo  get  Enqlish  articl*^s  when  l  A^SNGLISH  i&  not  available,  use 
NOT  LA^NONF.NGLISH. 


ERIC 


Probedures  No.  4 

Ul/7/85:> 

file:  sinBtruct 


PROCEDURES  FOR  DIALOG  SEARCHING 


ii     tarn  on  compu-ber  wi-th  disk  drive  open^ 

2,  When  the  dx&play  appears  on  the  screen,  enter  new  date  and  new  time  t if  t 
time  is  p.m.  use  military  time,  e.g.  14:00!)  with  a  carriage  return  after  each 
entry.     The  Main  Menu  display  will  now  appear  along  with  a  C>  prompt. 


3.  Turn  on  the  printer  and  make  sure  that  it  is  set  for  online  (the  top-fror 
''coggle  switch  must  be  pushed  up  so  that  both  red  lights  are  on>  .  Please  note 
if  at  any  time  the  printer  does  not  respond,   push  the  toggle  switch  to  online 

4.  At  the  C>  prompt,   enter  "project"  and  a  carriage  return.  Entering 
•'prdject**  selects  SMARTCOH  11  and  the  system  wi  11   respond  with  a  welcome 
message  and  ^instruct  you  to  strike  any  key  when  ready.     You  should  enter  a 
carraige  return-      The  system  will   respond  with  an  F>  prompt. 

5.  At  the  F>  prompt,   enter  "scom*'  and  a  carriage  return. 

6.  iviow  enter  a  7  without  a  carriage  return  to  change  the  printer  status  to 

<on;)  . 

7.  Next  enter  i    (begin  communication!)  i      Do  not  use  a  carriage  return^ 

Bm     Then  enter  an  O   (originate!).      Do  not  use  a  carriage  return. 

9.     Next  select  a  communication  network   in  the  following  order  of  preference 
(choose  either  Dialog  1   or  2,   whichever  you  use  most  1  requent ly!)  : 

Dialog  1  Dialoq  2 


DiAtNET  M  Q  with  carriage  return 

TEtENET  j  N  w/o  carriage  return 

TYMNET  k  O  w/D  carriage  return 

tJNINET  P  w/o  carriage  return 


10.  Enter  the  DIALOG  password  provided  lor  your  use  m  the  project. 

11.  Hit  the  Fl  key.  (The  Fl  key  is  in  the  top  left  hand  corner  of  the 
keyboard . !) 

12.  Next  enter  a  4   (receive  fiie!)-      Do  not  use  a  carriage  return. 

13.  Next  enter  a  2   (stop/start!)      Do  not  use  a  carriage  return. 
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14.  The  compater  will  now  ask  ^P^a  file  name^     the  ^isplay  lookB  iif??_^his5 

 .   Vou  should  enter  your_ searcher  numbers     For  example  if  your  searcher 

number  is  19,   you  would  enter  019  and  a  carriage  return. 

15.  Next  begin  the  file  as  assigned  with  a  b  file  number  command.  For 
example,   if   you  are  searching  ABl/lNFORn^   the  command  is  blB. 

16-     At  the  first  Dialog  prompt  ?,   enter  an  *  and  the  number  of  the  question 
ybu  are  currently  searching.     For  example,   *Q001 .      (.Ignore  any  invalid  code 
messages  that  will   result  in  Dialog  1 .  :i 

17.  Search  using  the  strategies  ybu  formulated  offline.  Type  as  many  of  the 
citations  as  you  need  to  refine  your  search  strategy.  When  the  desired  end 
result  set  is  obtained,  TYPE  the  set  out  in  FORMAT  1  (accession  number  onlyJ i 
Vbur  results  will  be  combined  later  with  those  of  other  searchers  and 
abstraccs  will  be  printed  but  for  the  combined  results.  For  example,  to  type 
out  set  number  3  with  20  documents  in  format  1,  the  command  would  be  t3/l/l-2 
if   you  are  using  Dialog  1   and  tS/l/all   if   you  are  using  Dialog  2. 

IS.     If  during  the  searching,  you  are  disconnected  f rbm  Dialog  and  find 
yourself    back  at  the  "scorn**  menu,   repeat  steps  11   through  13.     At  step  14, 
reenter  your  searcher  number  for  the  file  name.     Vou  will  see  the  following 
selections  on  the  screen:     File  exists  R(;e— enter.)  ,   EC  rase!)  ^   A<ppend>.  You 
should  select  A  for  append.      Proceed  to  step  IB,  etc. 

19.  Use  the  LOGOFF  HOLD  command   if   ybu  need  to  stop  your  search  temporarily. 

20.  To  begin  searching  the  next  question,  use  a  new  b  command  even 

if   the  next  question  is  to  be  searched  m  the  same  database  as  the  previous 
question. 

21.  At  the  first  new  Dialog  prompt  ?,   enter  the  new  question  number,  for 
example  #Q002. 

22.  When  all  assigned   questions  are  searched,  LOGOFF. 
2:3.     Hit  the  Fl  key. 

24.  Enter  jEf  (lend  communication.). 

25.  Enter  V   (yes).      This  is  the   last  step.      DO  NOT   fURN  OFF   THE  COMPUTER. 

2t>-  Attach  the  printout  of  your  search  results  to  the  Question  Request  Forms 
and  the  Search  Record  Forms  and  return  them  to  the  Project  Office. 

27.  The  checR  for  the  total  time  spent  searching  and  taking  the  tests  will  bi 
sent  to  you  later. 


ERIC 


Procedure  S 
files  record 


Procedures  lor  Recordxng 
Searches  from  Hard  Drive  onto  biskette 


li   the  computer  is  aii  n   turn  xt  on  with  the  dxsk  drive  open.     l*Jhen  the 
display  appears  on  the  screen^   enter  the  new  date  and  time  as  prompted.     Use  < 
carrxage  return  alter  each  entry i     il   you  have  a  C>  prompt,   enter  "project**- 
Vou  wxii  now  be  at  an  F>  prompt. 

2.  il   the  computer  has  been  left  on,   you  should  have  an  F>  prompt. 

3.  At  the  F>  prompt «  enter  ''format  A:/V"  and  a  carriage  return. 

4m     insert  an  unlc ^matted  1  loppy  dxsk  xnto  drive  A  and  lollow  the  prompt  to 
etrike  enters      the  system  will   respond  with  the  message  formatting ^ ^ ^ 
Mait  until  the  ^ystim  completes  formattxng  and  gxves  you  the  next  prompt 
for  a  volume  label 

Bm     For  volume  labels  enter  "S  _  _         using  the  number  of  the  searcher  who 
conducted  the  search^     For  example,   8019^     Use  a  carriage  return. 

6.  the  system  will  ask  xf  you  are  formatting  another.  Respond  with  Y  Ces>  o] 
N  <no^ .   Use  a  carrxage  returns 

7m     At  the  F>  prompt^   type  "dir"  and  a  carrxage  return  to  check  for  the 
presence  ol   the  searcher's  Ixle  on  the  hard  dxsk. 

8.     At  the  next  F>  prompt,   enter  the  following  command  for  the  first  file  to 
be  copied:     copy  .   _  .  A:   Use  a  carrxage  return. 

9i     At  the  next  F>  prompt,   snter  copy   _  _  _  G:   with  a  carriage  return. 

ISi     After  the  Ixrst  ixle  XB  copxed  onto  both  the  i loppy  and  the  6  sector  of 
the  hard  disk^   repeat  for  any  additional   files  to  be  copied. 

11.     Alter  the  last  ixle  xs  copxed,   turn  oil   the  computer. 
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Procedufes  No.  6 
C12/05/85> 
file:  sinstruct 


PROCEDURES  FOR  DIALOG  SEARCHING  USING  COMPAQ 


!•     Turn  on  cDmpu-ber  with  DOS  2»0  diskette  in  disk  drive  A. 

2-  When  the  display  appears  on  the  screen^  * 
time  is  p.m.  use  military  time^  e.g.  14:00)  with  a  carriage  return  after  eaoh 
entry. 

3.  Replace  the  DOS  2.0  diskette  in  drive  A  with  SMARTCOM  diskette. 

4.  Insert  the  data  diskette  provided  into  drive  B. 

S-  Turn  on  the  printer  and  make  sure  that  it  is  set  for  online  Cthe  top-fron 
^P€9^^  switch  must  be  pushed  up  so  that  both  red  lights  are  on).  Please  nct^e 
if  at  any  time  the  printer  does  not  respond^   push  the  toggle  switch  to  online 

6.     At  the  A>  prompt,  enter  ••project"  and  a  carriage  return.  Entering 
'•project**  selects  SMARTCOM  II  and  the  system  will   respond  with  a  welcome 
message  and  instruct  you  to  strike  any  key  when  ready.     You  should  enter  a 
carraige  return. 


7-     At  the  next  A>  prompt,   enter  •'scom^^  and  a  carriage  return.     Vou  will  now 
at  the  SMARTCOM  menu. 

Bm     Enter  a  B  without  a  carriage  return   <to  select  drive  m 

9p     Now  enter  a  7  wxthout  a  carrxage  return  to  change  the  printer  status  to 
COM)  . 

lb*     Next  enter  1    (begin  communication)^     Do  not  use  a  carriage  return^ 

ii .     Then  enter  an  O   (originate!).      Do  not  use  a  carriage  return, 

i2m  Next  select  a  communication  network  in  the  fol lowing  order  of  preference 
(choose  either  Dialog  1  or  2^   whichever  you  use  most  frequently!)  s 

Dialog  i  b± a 1 og  2 

DIAtNFT  M  Q  with  carriage  return 

TELENET  J  N  w/p  9^FF^^9^ 

TYMNET  k  b  w/o  carriage  return 

UMINET  P  w/o  carriage  return 

13.  Enter  the  DIALOG  password  provided  for  your  use  in  the  project. 

14.  Hit  the  Fl   key.      (The  Fl   key  is  in  the  top  left  hand  corner  of  the 
keyboard • ^ 

15i     Next  enter  a  4   (receive  fiie>.     Do  not  use  a  carriage  return. 
16 i     Next  enter  a  2   (stop/start)     Do  not  use  a  carriage  return i 


Procedure  No.   6  - 


Page  2 


17«     the  oompu-ber  wiH  now  as»;  for  a  file  name-     the  dispiay  iooks  like  thiss 

  You  should  enter  your  searcher  number.     For  example  if  your  searcher 

number  is  19,   you  would  enter  019  and  a  carriage?  return. 

18.  Next  begin  the  file  as  assigned  with  a  b  file  number  command.  For 
example,   if   you  are  searching  ABl/INFORN,   the  command  is  bl5. 

19.  At  the  first  Dialog  prompt  ?,   enter  an  *  and  the  number  of  the  question 
ydu  are  currently  searching.     For  example,   -it-QdQi  i      (ignore  any  invalid  code 
messages  that  will   result  in  Dialog  1.} 

20-     Search  Using  the  strategies  you  f prmUla^fa  of f lihe.__Type  a^ 
citations  as  you  need  to  refine  your  search  strategy^     (*Jhen  the  desired  end 
result  set  is  obtained^   TYPE  the  set  out  in  FORIiAT  i    (accessibh  number  ohiySi 
Y(5ur  results  will   be  combined   later  with  those  of  other  searchers  and 
abstracts  will   be  printed  but  for  the  combined  results.     For  example,   to  type 
out  set  number  3  with  20  documents  in  format  1,   the  command  would  be  €3/1/1-2: 
O 

if  you  are  using  Dialog  1   and  t3/l/ail   if  you  are  using  Dialog  2. 

21.     li   during  the  searching,   ybu  are  disconnected  f rbm  Dialog  and  find 
yourself   back  at  the  "scorn"  mehu^   repeat  steps  11  through  13.     At  step  14, 
reenter  your  searcher  number  for  the  file  name.     You  will   see  the  following 
selections  on  the  screen:     File  exists  RCe-enter) ,   ECrase) ,  A(ppend) .  You 
should  select  A  for  append.      Proceed  to  step  15^  etc. 

■  ~  U JE!~tTie  UDXjOn    I  lOLD  t^Ljiiiiiiarrtr  Xf~jjF'eu  >^eged~^p~  at       ynWip"  ri parch  t  cnmpap^P*  1  y  • 

23.  To  begin  searching  the  next  question,  use  a  hew  b  command  even 

if  the  next  question  is  to  be  searched  in  the  same  database  as  the  previous 
question. 

24.  At  the  first  new  Dialog  prompt  ?,   enter  the  new  question  number,  for 
example  #0002. 

25.  When  all   assigned  questions  are  searched^  LOGOFF. 

26.  Hit  the  Fl  key. 


28.  Enter  Y   Cyes!)  .      This  is  the   last  step.      DO  NOT  TURN  OFF  THE  COMPUTER. 

29.  Attach  the  prihtoUt  of  your  search  results  to  the  Ques;tipn  Request  Forms 
and  the  Search  Record  Forms  and  return  them  to  the  Project  Office. 

30.  The  check  for  the  total  time  spent  searching  and  taking  the  tests  will  b€ 
sent  to  you  later. 


Enter 


(!ehd  commUnicatioh J  . 
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Prpcedure  No»  7 
(12/05/85> 
file: record 


Procedures  for  Loading  COMPAQ 
Searches  from  Diskette  onto  iBM/XT  Rard  Drive 
and  -to  Create  Search  Diskette 


1.  if  the  computer  is  pf*^   turn  it  on  with  the  disk  drive  openi     When  the 
^??P^®y_^PP?^^^  date  time         prompted.     Use  a 
carriage  return  after  each  entry.     if   you  have  a  C!>  prompt,   enter  "project". 
Ybu  will  now  be  at  an  F>  prompt i 

2.  if  the  computer  has  been  left  on,   you  shouid  have  an  F>  prompt. 

3.  At  the  F>  prompt,   enter   "format  A:/V"  and  a  carriage  return. 

4.  insert  an  unformatted  f loppy  disk  into  drive  A  a       foiiow  the  prompt  to 
Btrike  enter.     the  system  wiii   respond  with  the  message  formatting... 

Wait  until  the  system  completes  formatting  and  gives  you  the  next  prompt 
for  a  volume  label. 

5.  For  volume  label,   enter  "S  _  _  _"  using  the  number  of  the  searcher  who 
conducted  the  search.     For  example,   SQi9i     Use  a  carriage  return. 

6.  The  system  will  ask  if  ybu  are  formatting  anothp?r.  Respond  with  Y  <esl)  or 
N   Cno!)  .   Use  a  carriage  return. 

7.  Insert  data  diskette  produced  on  the  COMPAQ  into  drive  A  of   IBM/XT i 

8.  At  the  F>  prompt,   enter:   copy  A:   _  _  _  G: 

9.  Remove  the  COMPAQ  data  diskette  from  drive  A  and  replace  it  with  the 
new,   formatted  diskette. 

10.  At  the  F>  prompt,   enter:   copy  6:    _  _   _  A: 

11.  After  the  last  file  is  copied,   turn  off  the  computer. 
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Procedure  No.  8 
(12/05/855 
file:  spro  jeotr 


Procedures  f  or  Project  Searches 

iNTRODUetiOivi 


A  project  staff  member  will  conduct  four  project  searches  for  each  question. 
A  staff   member  other  than  the  person  who  conducted  the  taped  interview  will 
conduct  these  four  searches.     Procedures  for  these  four  searches  are  described 
below. 


1.     Project  Search  #i    (Series  #1)   -  Search  of   Problem  Statement 


tisten  to  the  taped  interview  of  the  problem  statement.     Complete  the 
context  questionnaire  ^Form  9>.     Do  not  look  at  either  the  written  question 
statement   (Form  5>   or  the  user's  context  questionnaire   (Form  8). 

 .^^^.^PF'^.^?  to  record  the  search  strategy  you  formulate  offline. 

Record  the  time  spent  on  the  bottom  of   the  form.     At  the  top  of  Form  13  where 
it  asks  for  your  searcher  number,   use  your  number  preceded  by  a       "  (for 
example,  120!)  . 

 Conduct  the  search  online  using  Procedure  |nId.   4-     When  naming  the  file 

to  receive  the  data,   use  your  searcher  number  preceded  by  a  "1      (for  example, 
searcher  number  20  would  use  120!)  . 


2.     Project  Search  #2   (Series  #2:)   -  Search  of   Problem  Statement  with  Written 
Question  Statement 


 Read  the  written  question  statement  and ,   if  needed,   listen  to  the  tapec 

interview  again.     bsing  a  new  copy  of  Form  13,   formulate  a  searcR  strategy 
offline  for  the  question  based  on  the  information  contained  in  both  the 
written  question  and  the  interview.     Be  sure  to  note  the  amount  of  offline 
time  spent  on  the  formulation.     On  Form  13  where  it  asks  for  your  project 
searcher  number,   this  txme  use  your  number  preceded  by  a  "2*'   (for  example, 
220) 

Context  questionnaires  will  not  be  used   for  this  or  eit^ker  of  the 
remaining  two  searches. 
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3i     Project  Search  #3   (Series  #3:>   -  Search  at   Writ-ben  Question  Words 

Construct  a  search  strategy  based  only  on  the  words,  appearing  in  the 
written  question  statement   (Form  5?.     Agaih^   use  a  copy  of  Form  13  to 
record  this  strategy  and  the  time  spent  preparing  it.     Your  searcher  number  f< 
this  search  should  be  preceded  by  a  "3"   (for  eriample,  320!). 


4.      Project  Search  #4   (Series  #4)   -  Search  of  Written  Question  Words  Augmentec 
by  Thesaurus  Terminology 

Starting  with  the  words  appearing  in  the  written  question  statement 
which  you  selected  for  use  in  Project  Search  #3,   use  the  appropriate  thesaurus 
to  find  related  terminology.     Use  the  thesaurus  for  the  assigned  database.  li 
there  is  ho  jDrint  thesaurus^   use  the  online  thesaurus  if  available. 

Record  your  search  strategy  on  a  new  copy  of  Form  13.     For  searcher 
number  use  your  number  preceded  by  a  "A"   (for  example,  420). 


B-     Conduct  the  online  fearch  of   Project  Searches  #2^   #3,   #4.     Use  the 
procedures  outlined   in  Procedures  No.   4._   Create  a  new  file  for  each  of  these 
searches.     For  Search  ^2,   name  the  file  2  ~  - 

#3,  "        •*       3  ^  - 

#4,      "  V        "       4  _  I 

Fill   in  the  blanks  with  your  project  searcher  number. 
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Procedure  No.  9 
(4/21/865 
file:  proc9 


PROCEDURE  FOR  DETERMINING  SEARCH  TERM  OVERbAP 


1.  Start,  wi-th  the  5  listings  of  the  search  terms  used  by  each  searcher  for 
the  cMestibn  to  be  analyzed. 


EXAMPLE:  The  1 istings  for  question  001  searched  in  database  Oil  by  searchers 
002,  005,   013,  0169  and  021  would  include: 


DOgiOll .002 
pool Oil .005 
pool Oil .013 
D001011.016 
D001011.D21 


2.  Fill  in  the  Search  Term  Overlap  Form  with  your  name«  today's  dote  and  the 
question  number  and  searcher  numbers.  Note  the  time.  (When  you  are  finished 
fill   in  the  total  time  ybu  spent  on  that  question.!) 


EXAMPLE:.  For  the  listings  DOOlOll. 002,   D001011.005,   D001011.D13,  DOOIOII.DI6 
and  DD01011.021i  the  question  number  is  OOlOll  and  the  searcher  numbers  are 
002,   005,   013,   OlS,   and  021. 

The  form  should  be  filled  in  like  this:    (see  next  page> 
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SEARCH  TERM  OVERLAP 


NAME 


TODAY'S  DATE 
TIME  Sf=»ENT 


Questibh  Number  O 


  Ncimber  of 

Searcher     Search  Terms 

Number  Used  Search  Term  Overlap  with  Searcher  (Total  Number) 


_o  g_  > 
2.  £.  5 

k 


Q  0  5 

0  /  3 

O  /  & 

o  ^  t 

Form  IS 

/B&  >  f  i 1 e : over 1 ap 
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Procedure  —  page  3 


3.  itemize  £he  matching  search  ^errns  between  each  pair  of  searchers i  Fiii  i 
*h®_^?P_5®ll  ^®  described  using  "bhe 

iine  nambers^   from  ^he  lis-tings^  of  ^he  ma-tching  ^ermi     ££  will  be 
necessary  "bo  always  lisl^  ^he  line  numbers  in  correc't  searcher  number  orderi 
That  is  1^  when  comparing  searcher  002  with  searcher  005^  always  list  the  line 
number  of  the  term  used  by  searcher  002  first.     in  some  cases ,  a  term  has  bee 
added  to  or  crossed  off  the  listingi     Added  terms  should  be  matched;  deleted 
terms  should  be  ignored. 


RUtES  FOR  DETERMlNlNS  A  MATGH: 

Disregard  the  following: 

ii     a  leading  or  closing  parenthesis 

2m  the  contents  of  parentheses  used  in  phrases,  e^g.,  Cw)  Is  the  same  as  Cf) 
3i     limits,   eig.y  /de 

Count  the  following  as  a  match: 

ii     singular/plural   forms  of  the  same  word 

2i     truncations  of  the  same  word 

3i     verb  tenses  of  the  same  word 

4i     parts  of  speech  of  the  same  word 

Notes 

All  words  C3f  a  phrase  must  be  present  is  the  same  order  to  count  the  phrase  a 
a  matching  searc'i  term.     Follow  the  same  rules  as  just  described  above  for 
singular/plural  9  truncation,  tense  or  part  of  speech. 


4-     Fili   in  the  bottom  table  for  the  number  of  search  terms  used  by  each 
searcher  and  the  total  number  of  terms  which  overlap  between  each  pair  of 
searchers i     The  number  of  terms  used  by  each  searcher  is  the  line  number 
of  the  last  line  in  the  listing  for  that  searcher.     Please  note:  Even  if  ther 
is  a  search  term  added  or  corssed  off  the  list^   use  the  number  of  the  last 
line^     DO  NOT  ADD  OR  SUBTRAeT  FOR  THE  ADDED  OR  DELETED  TERM.      <The  program 
counted  these  and  the  number  appearing  on  the  last  line  i's  the  number 
app^iaring  in  the  final  data  analyses  for  the  40  questions.)     The  total  number 
matching  terms  for  a  pair  of  searchers  is  a  -total  of  the  itemized  matches 
recorded  in  the  top  ta  b 1 e  ^ 

EXAMPLE:     The  following  pages  show  descriptor  1 istings  DOOldll.002  and 
0001011.005  and  tiie  two  tables  filled  in  for  the  two  listings. 


522 


Listing  of  DOOiOll.002 


1  MIDDLE  AGE?  CW:)  CHILD? 

2  MlDDLEtlWJASE? 

3  MIDDLE  ASED/DE 

4  PARENT  CHiLD  eOMMbNi CATION/DE 

5  PARENT  ertiLD  RELATIONS 

6  PARTENTAL  ROLE 

7  MOTHER  CHILD  RELATIONS 

8  FATHER  CHlLD  RELATIONS 

9  MiDDLE<!iw:)ASED   tlWi)  CHILD? 
is  MIDDLE  ASEb  CHlLDREN/ID 

11  AGliMS  PARENTS/ID 

1 2  MI DDLE  <: W :i  AQED  <  W)  CHILDREN  / 1 D 

13  ASIMGCW.J  PARENTS/ID 


Listing  of  DbOlOll.OUB 

1   parent  child  communication 
2.  <.cj#ront-? 

3  geriat? ) 

4  parent  child  re] atibns!) 

B  interperBbnal   cbmmuni cations 

6  interpersphal  interaction 

7  parent  child  relations 
6  (tjj  dd  le  j  w.)  aged 

9  middlei  w>aged (w) children 

10  elderly  <.w.)  parent? 

11  older  <w.' parents.) 

12  adult  offspring 
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SEARCH  TERN  OVERLAP 


NAME 


TODAY'S  DATE 
TIME  SPENT  . 


Oaestion  Number 


0  O  I     0  1  I 


Searcher 
Number 


Search  Term  Overlap  wl'th  Searcher  (Itemized) 


Number  of 
Searcher     Search  Terms 

Number  Used  Search  Term  Overlap  with  Searcher  CTotal  Number) 


_o  ^  ^ 

L 


^  ^  £ 

P.  L  fe. 

Form  16 

<4/21 /S6  >  f  i  i  e  i  over i  ap 


PRbCEbURE  IQ  (3/27/86) 


PROCEbLiRES  FOR  CdMPLETiNS  QUEStiON  REQUEST  FORMS 
USING  THE  iBM/XT  COMPtJTER 


Turn  pn  the  ±BM/Kt  computer,  using  the  switch  on  right  side  of  the 
computer • 


2m     When  the  computer  prompts  you  for  the  date  and  time^  enter 

these  using  the  format  4/1/86  and  military  time  in  hours  and  minutes. 

3.     Put  the  QUESTF0RM5A  diskette  in  Drive  A- 

4»     At  the  C>  prompt^   enter  copy  arquesti^K-  c: 

Should  get  response  that  all  QUEST  files  are  copied. 

5.  Replace  the  QUESTF0RM5A  diskette  in  drive  A  with  PCWRiTE2i 

6.  At  the  C>  prompt^  enter  the  foiiowing  command     ed  quest. qqq  where  qqq 
represents  the  question  number  you  will   be  entering  into  the  database; 

7.  At  the  next  screen  from  P€WRiTE,   press  the  Fl   function  key  on  the  left  si 
of  the  keyboard; 

8.  You  should  now  have  a  screen  of  the  form  on  which  you  will  enter  data. 
9^     Question  Number  should  already  be  entered  for  you. 

Id.   Find  this  question  number  in  the  first  column     of   the  list  entitled 
''Assignment  of  Questions  to  Searchers". 

11  i   P^?D^_t^^_bB  number  in  the  fourth  column  of  this  list,  which  corresponds  t( 
the  question  number • 

12i   Enter  this  DB  number  on  the  line  on  your  screen  entitiedj  biAt06  DATABASE 
USED.   Enter  a  hyphen  after  the  DB  number  and   look  up  this  number  on  the 
list  entitled:    "DATABASE   (supplier)   and  enter  the  database  name  (portion 
in  capitals!)  • 

13.  The  remaining  data  for  input  is  obtained  from  Form  5^  entitled  "Question 
Request  Form"..   Please  check  with  Alice  or  Donna  if   you  are  not  sure  about 
the  spelling  of  words  as  written  or  if  you  need  clarification  of   any  kind 

14.  Item  Al .   Enter  the  brief  title  as  written. 

15.  Item  A2.   Enter  the  question  statement  as  written. 

16-   Item  Bm   If  3  is  circled,   enter   "  precise";   if  4  is  circled^   enter  "  broad 

17.  Item  C.   Enter  the  word  or  phrase  following  the  number  circled^ 

18.  Item  D.   If  11   is  ciz?cled^   enter  "English  only";   if  12  is  circled^  enter 
"any  language"  and  the  languages  stated,   if  any;   if   both  11  and  12  are 
circled,   then  enter  the  in^qiC'Aatioh  for  both. 


»  Item  E.  If  13  is  circled,  enter  "  Last  5  years"  ;  if  14  is  circled^  entei 
JC  '-^i?**^®***  f.'         is  bircled^  enter  the  years  specified. 
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PROCEDURES  FOR  eOMPtETiNS  QtJESTiON  REQUEST  FORMS 
USiNS  THE  IBM/XT  COMPUTER 


20.  i'tem  F«  Enter  any  of  the  items,   compieted,   unless  crossed  off. 

21.  Check  all  of  the  items  you  have  entered  on  the  screen.  Correct  any  errors 
you  have  made. 

22 i  After  all  the  corrections  are  made,  press  the  function  key  Fl  followed  by 
F2i 

23 i   Turn  on  the  printer^   if   it  is  not  bn^   using  the  switch  on  the  left  side  of 
the  printer i   If   both  red   lights  are  not  on^   then  also  toggle  the 
third  button  from  the  bottom^   forward  towards  the  label   "  on!ine'*i  This 
prepares  the  printer     for  printing i   Rol 1  the  paper  forward  until  the  print 
head  is  Just  b&low  the  perforation i 

24.  At  the  C>  prompt^   enter  pr  quest iqqq^   where  qqq  is  the  question  number 
you  have  just  inputs 

25.  You  will  be  asked  some  questions^  press  the  return  key  at  the  first  pause 
and  the  esc  (escape)  key  at  the  second  pause.  A  copy  of  the  data  you  have 
Just  input  will   be  printed i   Check  this  over  for  any  errors i 

26.  If  there  are  errors,   go  back  to  step  6  and  re--ehter  the  same  question 
number  and  make  the  corrections  needed.   Then  go  to  step  22  and  print  a  new 
copy  of  the  printout  using  steps  24  and  25.   Give  copies  of  these  printouts 
to  Donna  or  Alice  at  the  end  of  the  morning  and  afternoon  sessions. 

27.  If  there  are  no  errors,  go  to  step  6  and  enter  the  file  name  for  the  next 
question  number  you  will  work  on. 


PROCEDURE  11  <6/3/a6^ 
F=»F;jCEDURES  for  using  DBASE  files  on  eOMPAQ  eOMPUtER 


1. 

2. 
4. 


Insert  the  DBase  III  diskette  in  drive  A. 


Insert  the  appropriate  DBase  Files  Diskette  in  drive  B. 

See  listing  of   file  contents  of  Diskettes  I  and  II  in  item  14. 

At  the  A>  prompt,   type  dbase. 

The  DBase  ill  will   load  and  give  you  a  set  of  options  to  use. 
These  procedures  are  based  on  selecting  none  of  these  options  but 
in  entering  the  appropriate  DBASE  commands  at  the   .   prompt.  These 
commands  are  described  in  detail   in     the  DBASE  Manual  and  briefly 
summarised  in  following  items. 


•set  default  to  b: 

This  command  is  used  to  tell  the  program  that  the  DBase  Files  will  be 
found  in  drive  bs 

This  command  is  used  to  tell  the  program  the  name  of  the  file  you  want 
to  work  with.   For  a  directory  of  the  DBase  files  on  a  diskette,   enter  tt 
command   .dir  and  a  listing  of  ail  the  files  suffixed  with   .dbf  will  be 
listed  m 


•set  print  on 

if  you  want  to  print  while  in  the  dbase  files,   first  turn  on  the  printei 
in  the  usual  way  with  the  on  switch  and  online  button,   then  while  in 
dbase,  use  the  above  command.   This  will   print  the  commands  and  the  file 
contents  if  the  .list  command  is  used.   It  will  not  print  any  of  the 
contents  if  the   .browse  command  is  used. 


Si        iset  print  off 

This  command  turns  off  the  printer  while  in  dbase. 

9.  .list  

This  command  produces  a   list  of  all   items  in  the  file.   The  items  will 
scrbll  without  any  breaks.   This  will   be  printed  if  the  .set  print  on 
command  has  been  issued.   Records  oannot  be  changed  using 
this  commands 


ID.  .browse 

This  command  lists  records,  one  screen  at  a  time.   The  fields  are  listed 

horizontally  ^^^changes  can  be  made  to  the  fields  as  needed.   To  move  to 

the  next  screen  or  next  fields,  use  the  appropriate  cursors.   When  you  rea 
the  end  of  the  file^   the  program  will  ask  you  if  you  want  to  add  new 
records.   Answering  yes,  allows  you  to  add  new  records.  To  get  back  to  the 
dot  command  prompt  use  the  esc  key • 

11.  .edit 

This  command  lists  records  one  at  a  time  in  an  editing  format  to  enter  an 
changes  heeded if  a  specif ic  record  is  wanted,  the  record  number  should 
follow  this  command^   for  example  .edit  3.   Upon  reaching  the  last  record  i 
the  file,  the  program  will  ask  you  if  you  want  to  add  new  records. 
Answering  yesj   allows  you  to  add  new  records.   To  get  back  to  the 
dot  command  prompt  use  the  esc  key . 
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PROCEDURES  FOR  USING  DBASE  FILES  ON  COMPAQ  COMPUTER 
•ciose  databases 

This  command  is  used  tc  close  the  file  or  database  currently  active^ 
before  using  another  file.  Several  databases  can  be  in  use,   but  it  is 
safer  to  close  each  database  after  using  it^  if   it  is  r.o  longer  needed. 
This  command  must  be  used  before  exiting  from  dbase. 

•  quit 

This  command  is  used  to  get  out  of  the  DBase  programiNot^  that  ail  files 
should  be  closed  using  the   .ciose  databases  command  before  using  the  .quii 
command • 


14 Contents  of   DBase  Diskette  i  &  il 

Note  —  fp^ff*  numbers  used  for  the  file  name  correspond  to  the  printed 

forrn  numbers,   from  which  the  data  was  entered.   In  some  cases,   the  files  at 

incomplete  if  subsequent  data  was  not  needed. 


DISKETTE  I 

FILE  NAME 
FORMS . DBF 

TIMES. DBF 

TIME31 . DBF 

DISKETTE  II 
FORMS. DBF 
F0RM7_S -  DBF 

FORMii .DBF 
F0RM16.bBF 
SCORE. DBF 


FILE  CONTENTS  

bist  of  questions,   database^   searcher  codes  and  brief  titles 

Searcher''  s  .  preferred  time  for  performing  searches 

tist  of   preferred  search  times,    listed  in  order  for  each 
searcher 


tist  of   searcher  information,   coded  from  form  3 

Scheduiing  of  interviews  for  users  and  project  searchers  and 
other  ^information  from  forms  7  and  8 

Question  analysis  and  classification  data  from  form  11 
Search  term  overlap  data  from  form  16 
Cognitive  test  scores  for  ail  searchers 


SCONTEXT.DBF     Context  values  f or  pro ject  searchers  on  form  9  and 
outside  searchers  on  form  10. 
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B>Nb'bes  on  DBase  Commands 
Chapter  I 

DBase  COMMANDS  -  GENERAL 

a>dBase 

-•command 

-.set  default  to  B:    <f loppy) 
-.set  default  to  C:    Chard  disk) 

clear  (clears  screen) 
—.quit  (qets  you  out  of  Dbase) 
-.dir   (list  of  files) 

-•USB  file  name,  without  extension  <get  file  for  use) 
-.display  (displays  field  names  as  headings  in  table) 
-•display  all    (displays  one  screen  at  a  time) 
-.display  next  n  (#  of  records  to  be  displayed) 
—.display  record  n  (to  display  a  specif ic  record) 

-.goto  n  or  -.go  n  cr  -.n  followed  by  -.display  (to  display  a  specific  rec 
also) 

-.list  (displays  all  records  as_ a  table) 

-.list  structure  (lists  all  fields  in  database  and  length) 

Commands  can  be  shbrtened  to  4  letters;   field  names  10  characters; 
max  128  fields;  up  to  ID  database  files  open  at  once-. 


Chapter  II 


ADDING  RECORDS 
—.help  command 

esc  tc  get  back,   pg  up  to  previous  screen,   pg  dn  to  next  screen 
—.set  menu  on  (displays  the  control  keys) 
—-append   (to  add  new  records) 

automatically  in  the  overwrite  mode 

for  insert  mode,   press  ins  key  at  insert  location  &  press  again  to  stop 

use  del   key  or  backspace  to  delete  characters 
—.edit  n   (to  correct  existing  record  n) 

going  to  next  record  saves  changes  automatical ly 

to  save  changes  for  this  record  bnly,^  use  ctrl  end; 

to  cancel  the  changes  for  this  record  only,  use  esc 
"•Q?.**??  ^  to  get  to  beginning  of  file) 

-.browse  (  to  view  all   records  ;   can  use  edit,   append  or  insert) 
use  Ctrl  home  to  see  options  in  browse  mode 

Ctrl   rt  arrow  &  ctrl   left  arrow  to  browse  to  right  or  left  of  screen 
DELETiNS  RECORDS 

—.delete  record  n   (marks  record  for  deletion) 

to  make  invisible  for  calculations,   use  command  -iset  deleted  on; 

to  make  visible  again  -.set  deleted  off;   to  recall   records  -.recall  all 

to  remove  permanently,   follow  delete  command  with  command  -.pack 
•-.edit  n  follow&d  by  ctrl  u  and  ctrl  end  marks  for  deletion,   ctrl  u 

puts  record  back  in 

SEtECTiNS  FiEtbS  FROM  A  DATABASE  - 

  5^9 

-.display  or  -.list  or  -.browse  fields  (then  list  fields  of  interest) 
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SETTING  UP  DBASE  FILES 

Nos^  efficient  if  have  16-20  fields  in  1  file  and  set  up  more  files  if  need 
Five  "types  of  fields 

C  -  csharacter  text  up  to  2S4  characters 

n      liemb  kept  in  separate  disk  &  up  to  4000  characters 
N  -  Numeric       numbers,p  decimals,   plus^   minus,   no  commas 
L  -  1  character  —  true/false,  yes/nb 
D  -  date  8  spaces  store  as  Julian  date 

CREATING  FILE 

—•clear  all    Cto  close  open  files) 
—•assist   ^provides  a  help  screen) 
options 

select  set  up  to  create  a  new  file 

select  use  to  modify  an  existing  database  or  index 
prompts  for  file  name 

prompts  for  field  name,   field  type,   width  of   field,   decimal  (?) 
when  finished  entering  fields,   ctrl  end  and  return 

to  enter  memo  field,  ctrl  home  puts  into  test  editor  and  can  enter  2  scri 
of  notes  and  ctrl  end  to  return  to  record 

RETRIEVING  DATA 

—•clear  all 
—•assist 
options 

select  use  to  view  an  existing  database 
prompts  for  disk  drive  a,   b,  c 

shows  directory  of   files,   select  file  of  interest 
select  retrieve 

select  option  of   interest,   display,   sum,   average,   count,    label,  report, 
position 
—.quit  to  end  session 

CHAPTER  IV 
SELECTING  RECORDS 

RELATIONAL  AND  LOGICAL  OPERATORS 

Relational  operators   

-.DISPLAY  FIELDS  FOR  FIELD  CONDITIONS 
first  set  of   fields  specifies  fields  to  display 

for  clause  denotes  what  you  field  yoU  want  to  specify  relation  conditions 
conditions  include  =  <   >  or  combinations  of  these 

can  use  tcj  find  numeric  or  characters  in  all  fields  except  memo  or  logics 
fields—.  However  to  find  characters  type,  the  characters  must  be  enclose 
in  single  or  double  quotes—.  Note  that  must  specify  if  want  caps  or  loWei 
case  because  caps  have  higher  value  than  lower  case-. 

Logical  operators 

conditions  are  specified  as  -.AND-.   -;6r-;  -.NOT-. 
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These  can  be  combined  with  the  relational  pperatorB-. 

and  is  processed  before  the  brt  use  parentheses  for  the  logical  operators 
needed 

If  ybu  want  to  save  data  you  can  store  it  as  a  memory  variable,  which  is  like 
field  riamef 

store  data  to  memory  variable 
-iciear 

—•display  memory  (lists  variable  name  &  type  S  data) 
now  you  can  use  the  memory  variable  name  in  place  of  the  data 
If  you  want  to  save  the  memory  variables  for  use  later 
save  to  constant 
-^release  ail 

—i display  memory  , 
to  get  them  back 

—  i restore  from  constant 
-.display  memory 
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i.  Introduction  arid  List  of  Abbreviations  and  Codes  Used  in  this  Report 

ii.  Brief  Outline  of  ProgriPifns  in  Operatibnal  Order 
III.     Program  Descriptions 


A. 

STEP6 

B. 

STEP? 

C. 

STEPl  and 

STEP2 

D. 

STEPS  and 

STEPS 

E. 

STEP4  and 

STEP9 

F. 

STEPS  and 

CFLAG 

G. 

STEPl 1 

H. 

STEPl 0  and  STEPl 2 

IV.  step-by-step  Procedures  for  f?anning  Programs 

Aa     Procedures  for  Processing  Diaibg  Transcripts 

B.     Procedures  for  Processing  Evaluated  Abstracts  and  for  Producing 
Final  Data  Summaries 

V.  Files  Used  and  Created  by  Each  Program 
Vi.       Files  Categorized  by  File  Content 
Vii.     File- Storage 


VIII.  Examples  of  Files 

A.  S039.TXT 

B.  S029tD08.039 

C.  S0290be.039 
p.  C029db8.C339 

E.  T029008.039 

F.  Tb29008.039 

G.  0029008.039 

H.  Q029C308.bAT 

I.  Q029008.c6D 
J.  F02?tD08.ABS 
K.  F029b08.b39 
L .  F029008 . DAT 
M.  Qb29008.bUT 
N.  pb29bd8.b39 
O.  SEARb39.bjJT 
P.  QUESb29.bUT 

files;;  title  pagei  dpr.tc;  ±i  dprsint;  li*  dpr^but?  III.  dpr.pro; 
IViA;  cjpr;run;  IV. B.  dpr.ran?  V.  dpr.lisj  VI.  dpr.catj  VII.  dpr.sto; 
yiii.  dpr.eHa 

Uf.  -'  ■ 

■  ■.  ^  ^  X  i< 


(Unedited) 
(Edited) 

(Online) 
(Total) 
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111.   PROGRAM  DESCRIPTIONS 


StEP6 

This  program  was  designed  to  edit,  the  Dialog  online  transcript  file 
produced  using  Smartcom  by  each  searcher.     The  online  transcript  was  recorded 
on  diskette  at  the  time  of  the  actual  sear(5h.     A  printed  c5opy  of  the  transcrip 
was  also  recorded  at  the  time  of  the  original  search.     The  transcript  recordec 
on  the  diskette  included  every  key  stroke  the  searcher  used.     Typos  were 
included i     It  also  included  all  Dialog  system  responses  to  every  command  used 
by  the  searcher.     Searchers  were  instructed  to  search  as  they  normally  would. 
They  were  not,  however ,   instructed  to  download ,   or  print  out  abstracts  as 
answers  to  the  questions  they  searched f  except  as  they  heeded  to  see  the 
retrieval   results  they  produced i     Instead^   searchers  were  instructed  to  produc 
their  final  answers  in  Dialog  format  1,   accession  number  ohlyi     bater  these 
accession  number  sets  were  processed  by  project  staff  and  abstract  sets 
produced . 

  The  purpose  of  STEP6  was  to  edit  or  select  from  the  complete 

transcript  those  portions  of  the  transcript  desired  for  further  study.  These 
portions  of  the  transcript  were  identified  as  i>     all  commands  used  by  the 
searcher,  e.g.   select  commands,  type  commands,   begin  commands,  etc^     2>  Dialc 
accession  numbers  produced  by  all  type  commands     3>     the  cost  and  time 
spent  in  each  Dialog  file  used^ 


The  search  transcript  on  diskett     was  named  the  "sss"  file,  where  "sss 

was  the  number  assigned  to  the  searcher^  STEP6  read  in  an  "sss"  file  and 

created  an  S^sss". TXT  file  for  that  searcher .     The  S"sss" .TxT  file  included 
only  those  portions  of  the  "sss"  file  mentioned  above. 
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tii.B.  STEP? 

The  pcirpose  of  STEP?  was  td  further  drganize  the  search  transcript 
according  to  the  intended  use  for  each  part*     The  commands  used  by  the 
searcher  were  analyzed  for  strategy  and  search  terms  used»     The  accession 
numbers  of  the  documents  retrieved  were  needed  to  produce  a  set  of  abstracts, 
the  cost  and  time  information  was  used  for  records  of  time  spent. 

  STEP?  read  in  the  S"sss"iTXT  file  and  separated  it  into  three  sets 

of  files  for  each  question  the  searcher  did.     The  three  files  for  each  questic 
inciudedi  1)       an  S"qqqbbb"i^"sss"  file  of  type  comrnands  and  accession  numbers 
2)     a  e"qqqbbb"i'^sss"  file  of  commands  used     3S     a  T**qqqbbb'^''sss''  file  of 
time  spent  and  cost i     in  this  case ^"qqqbbb"  represents  the  question  number/ 
database  number i     The  number  of  files  produced  by  STEP?  for  the  searcher  text 
file  was  3   (one  for  each  S^qqqbbb" .  "sss"  e"qqqbbb" .  "sss"  and  T'^qqqbbb"  i  ••sss*'> 
times  the  number  of  questions  searched. 

So,   for  example^   if  searcher  001  did  6  questibnsj   from  the  S001.TXT 
file,  STEP?  would  produce  one  S**qqqbbb"  .001 ,  one  C"qqqbbb**.ODl  and  one 
T"qqqbbb" .001   file  for  each  of  the  6  questions,   for  a  total   of  18  files. 
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III.C.  STEPl  and  STEP2 

THe  purpose  of  STEP!  ahd^  in  part,  STEP2  was  tb  create  master  lists  oi 
the  question/database  nurhbe       and  the  searchers  who  did  each  question.  The 
list  of  searcher  who  did  a  particular  question  was  used   later  in  subsequest 
programs  as  a  prbmptihq  and  record  Reepinq  mechanism.     The  other  purpose  of 
STEP2  was  to  create  sorted  and  merged  sets  of  accession  numbers.     Each  prograc 
will  be  described  individually. 


STEPl 

 This  program  produced  a  master  list  of  question/database  numbers  callc 

the  TOTAL-LST  file.     As  a  qUestidn/database  number  wa^  to  the  TOTAL. LST 

file,   the  program  also  named  a  Q"qqqbbb" . LST  file  for  that  question. 


STEP2 

SfEP2  had  two  functions.     The  first  function  was  tp  sort  and  merge 
sets  of  accession  numbers  which  were  read  in.     Because  c)ach  question  was 
searched  9  times,  the  accession  numbers  produced  by  sac::^^;  individual  search  of 
a  particular  question  had  to  be  sorted  and  then  merged,  eliminating 
duplicates,   so  that  one  set  of  accession  nLlmbers,   representing  the  combined 
results  of  all   9  searches,   couJd  be  uploaded  to  Dialog  and  an  abstract  set 
downloaded. 

the  second  function  of  sfEP2  was  tb  fill   in  the  Q"qqqbbb".LSf  file 
with  the  searcher  numbers  of   all   bf  the  searchers  who  did  that  question. 


For  the  question  being  processed,   STEP2,   as  instructed  by  the  cypGV^tcyt 
read  in  each  S"qqqbbb" . "sss"  file  bfaccessibn  numbers  retrieved   by  each 
searcher  who  did  the  question.     The  9  S"qqqbbb" . "sss"  files  for  the  question 
were  each  sorted  individually.     The  sbrted  sets  were  named  0"qqqbbb" . "sss" 
files.     The  9  Q''qqqbbb'' .  "sss"  f  i  les  were  then  m  bne  set  of  sbrted 

accessibn  numbers  called  the  Q"qqqbbb" . DAT  file  fbr  the  question.  This  was  th 
file  that  was  subsequently  used  to  produce  the  abstract  set. 


In  addition,   as  each  searcher's  accessibn  numt3er  set  was  read  in^  the 
searcher's  number  was  added  to  the  the  Q"qqqbbb" .LST  file  for  future  use. 
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III.D.  STEP3  and  STEPS 

The  purpose  of  STEP3  and  STEPS  was  to  prodace  sets  of  abstract 
in  a  form  appropriate  to  send  to  the  Qsers  who  submitted  the  AO  research 
questions.     STEPS  was  used  to  create  a  set  of  accession  numbers  that  Dialog 
could  read.     STEP5  was  used  to  add  a  relevance  judgement  blank  to  the 
down 1 oaded  abstracts • 


STEPS 

In  order  to  obtain  actual  abstracts  from  the  Dialog  system  (searchers 
did  not  do  this;  they  only  produced  accession  number  sets)     the  merged 

set  of  accession  numbers  for  each  question  had  to  be  uploaded  to  Dialogs^  

STEPS  was  used  to  prepare  the  succession  nurnber  sets  for  this  purpose i  STEPS  

processed  the  merged  accession  number  set  for  each  question  and  produced  a  set 
of  accession  numbers  which  included  a  Dialog  "keep"  command  for  each  individua 
accessibh  number  in  the  merged  set.     STEPS  read  in  the  Q''qqqbbb"iDAT  file  for 
the  question  and  created  a  Q'*qqqbbb''^eOD  file  for  the  questioni     This  is  the 
file  that  was  then  uploaded  to  Dialog.     The  abstracts  were  then  downloaded  int 
a  file  named  Q"qqqbbb" . ABS. 


STEP5 

The  downloaded  abstract  file  for  each  question^  named  the 
Q"qqqbbb" . ABS  file,  was  then  processed  using  STEPS.     This  program  inserted  a 
blank  form  after  each  abstract  in  the  Q"qqqbbb"iABSf  form  was 

provided  to  the  user  for  his  or  her  evaluation  of  each  abstract.     The  blank 
form  looked   like  this: 

RELEVANT   PARTiAttY  RELEVANT   NiDT  RELEVANT  

the  file  created  by  STEPS  was  named  the  F^qqqbbb" . ABS  file^     This  file  was 
printed  on  two-part  form  and  mailed  tij  the  user. 
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iii.E.  STEP4  and  STEP9 

The  purpose  of  -these  two  progzrams  was  to  pzrbcess  the  evaluated 
abstracts i     These  programs  assigned  the  zrelevance  judgements  about  each 
abstract  to  the  accession  number  repxresentirig  that  abstract i     This  relevance 
information  was  used  as  final  data  about  the  question  searched  and  the 
searchers  who  performed  the  search. 


STEP4 

For  the  question  being  processed,   STEP4  wrote  out,   in  turn,  each 
accession  number  from  the  merged  accession  number  set  in  the  Q"qqqbbb" . DAT 
filei     As  the  individual  accession  number  was  displayed,   the  operator, 
using  the  printed  set  of  abstracts  with  evaluations  returned  from  the  user, 
input  the  relevance  judgement. 

After  all  of  the  individual   judgements  for  each  abstract  in  the  set  foi 
the  question  were  inputs   the  ihformatif^n  was  recorded   in  a  file  named 
F''qqqbbb**«DATi     The  F"qqqbbb" .  DAT  file  included  each  of  the  accession  numbers 
from  the  Q''qqqbbb'' i  DAT  file^   but  included  additionally  for  each  accession 
number  in  the  file  a  code  for  the  specific  relevance  Judgement  returned  about 
the  corresponding  abstract. 

STEP4  also  produced  a  file  named  Q"qqqbbb" .OUT.     This  file  was  a  list 
of  each  accession  number  in  the  set  for  the  question  with  the  words,  RELEVANT, 
PARTlAbbV  RELEVANT,   NOT  RELEVANT  and  NOT  EVALUATED  following  each  accession 
number.     This  file^   when  printed  out,   was  used  to  check  the  accuracy  of  the 
input  of  the  relevance  Judgements. 


STEP9 

STEP9  was  used  both  to  edit  relevance  Judgements  and  to  assign  the 
relevance  Judgements  back  to  the  individual  searchers  who  searched  the 
question •     if  an  error  was  found  in  the  relevance  Judgement  inputs  STEP9 
allowed  the  operator  to  ^r^lt  the  relevance  Judgement  supplied.     After  the 
relevance  Judgements  were  checked  and  found  to  be  accurate,  STEP9 
was  used  to  assign  the  relevance  Judgements  back  to  the  accession  numbers 
produced  by  each  individual   searcher  of  the  question. 

STEP9  used  the  F*'qqqbbb" .  DAT  file  and  compared  each  acbessibh  number  if 
the  file  with  the  accession  number  in  the  individual   searcher^  s  Q'*qqqbbb" .  "sss" 
fiie^     The  relevance  Judgements  for  the  question  as  a  whole,   as  represented  in 
the  F^qqqbbb'' i  DAT  file^   were  then  assigned  back  to  the  accession  number  sets  of 
the  9  searcher's  who  performed  the  searches  of  the  question.     This  information 
^as  stored  in  9  F"qqqbbb" . "sss"  filesi 
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iil-F-  STEPS 

The  purpose  of  STEPS  was  to  calculate  the  time  spent  by  each  Gearcher 
to  do  each  question  he  or  she  was  assigned.     The  total  amount  of  time  spent 
by  each  searcher  doing  each  question  was  calculated  by  adding  "the  time  he  or 
she  spent  online  to  the  time  he  or  she  spent  offline  preparing  to  go  chline. 
Each  of  these  amounts  of  time  was  recorded  and  the  data  was  included  in  the 
final  data  about  each  question  and  each  searcher. 

The  amount  of  time  spent  online  for  each  search  was  caF3tured  during  thi 
original  online  sessioni     The  online  time  was  stored  in  the  T"qqqbbb" i "ess" 
file  by  the  program  STEP? i     STEPS  was  used  to  add  the  offline  time  to  the 
online  time^   record  each  value^   and  calculate  a  value  for  the  total   time  spent 
by  each  searcher  on  each  question.     Using  the  Q''qqqbbb''.bST  file  for  the 
question  as  knowledge  about  which  searchers  did  the  question,   STEPS  promptedy 
in  turn,   for  a  time  spent  offline  by  each  searcher  listed  in  the  Q''qqqbbb'' .bST 
file.     The  time  offline^   time  online,   and  the  total  time  were  then  stored  in  a 
revised  version  of  the  T"qqqbbb'' ^  "sss"  file^ 

CFLAG 

This  program  was  used  to  edit  the  Q"qqqbbb" .LST  file.     It  was,  in 
essence,   actually  a  subroutine  of  the  program  STEPS.     Because  it  was  added 
later,    it  became  a  separate  program.     The  purpose  of   CFLAG  was  to  enable 
STEPS  to  precede  anew  if   the  first  run  of  the  program  failed  to  read  the 
T"qqqbbb". "sss"  file  accurately. 
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IIl.S.  StEPll 

STEPll  was  designed  "to  determine  search  §erms  used  by  each  searchei 

for  each  ques'tion  searched*     The  program  select^ed  t^hese  search  t;erms  fron  -the 
command  statements  used  by  the  searcher - 

-.-The  commands  used  by  each  searcher  to  search  each  question  were  stored 
in  the  C^*qqqbbb"."sss*V  f  ile  f or  that  question  "qqqbbb'*  and  that  searcher  ^*sss"i 
This  file  was  read-in  to  STEPll-     The  jDrogram  then  selected  but  the  search 
terms  used.     Essentially^   STEPll   ignored  the  actual  commands  such  as  ''ss**  or 
•'c"  or  "t",  the  set  numbers  such  as  "s3"  or  "sA",   and  the  operators  such  as 
•'and'V  or  "or arid  wrote  the  resulting  words  as  a  list  of  search  terms  into  a 
file  called  b"qc}qbbb" .  ••sss" .     STEPll  also  counted  the  total  number  of  search 
terms  .found  in  each  C"qqqbbb" .  "sss"  file.     This  value  was  then  used  as  final 
data  in  the  final  question  data  and  final   searcher  data  files. 
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ill.H.  STEPlb  and  STEPi2 

Thevpurpbse  of  'these  -two  prbgrams  was  £0  bumula'te  'the'c'B'ta  produced  by 
the  other  Pt'ogr'sSs  into  final  data  files-     The  final  data  was  presented  in  tuo 
different  cbmolations  of  essentially  the  same  data^     A  final  data  file  was 
prepared  aboctt  each  searcher  by  STEPiOi     A  final  data  file  was  prepared  aboti£ 
each  question  by  3TEPi2i     The  final  searcher  data  file  was  itemized  by  each 
question  he  or  she  searched.     The  final  question  data  file  was  itemized  by  eaci 
searcher  who  searched  that  question^ 

  Both  STEPIQ  and  STEP12  used  the  same  data  files  as  inputs     These  files 

included  the  evaluated  accession  numbers  for  each  searcher  who  did  the 
question^   storad  in  the  9  F"qqqbbb" • "sss"  files,  the  commands  and  search  terms 
used^   stored  in  the  9  e"qqqbBb"  i  *'sss*'  files  and  the  9  D"qqqbbb"  ^ '^sss*'  files, 
and  the  time  spent  ^   stored  in  the  9  T*'qqqbbb*' i ''sss"  files.     STEPi2  used 
additionally  the  merg>ed  set  of  evaluated  accession  numbers  for  the  question, 
stored  in  the  F*'qqqbbb"  i  DAT  files 

STEPiQ  produced ^  using  an  updating  process,   a  final   report  about  each 
individual  searchers     As  each  question  was  completed^   the  information  about  thi 
question  was  added  as  an  update  to  the  searcher  final  data  files  for  those 
searchers  who  did  the  Qttestioni     T^e  fianl   searcher  data  file  was  named  the 
SEAR"sss**.OUT-     In  this  case,   as  before,   "sss''  represented  the  number 
assigned  to  that  searcher. 

STEP12  produced  a  final  report  about  the  individual  question  in  a  file 
called  the  QUES^qqq" .OUT.  Here,  "qqq**  represented  the  number  assigned  to  that 
questions 
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IV. A.   PROCEDURES  FOR  PROCESSING  DIALOG  TRANSCRIPTS 


The  Dialog  eeardh  Bession  was  cap-tured  ph  di&ket.t.e  and  on  a  print.-ou-t. 

The  procedure  described  here  includes  all  processing  of  the  Dialog  t^ranscript 

t^hrouQh  the  production  of  an  abstract  set  which  was  sent  to  the  user. 

A  second  procedure  titled,   "Procedure  for  Processing  Evaluated  Abstracts" 

describes  the  second  half  of  the  total  data  processing  procedure. 


The  procedure  for  processing  the  Dialog  transcripts  foollowss 


1.     Copy  the  search  transcript  created  on  a  diskette  into  the  /xt  drive 
containing  the  STEF.EXE  programs.     In  this  case,  in  order  to  access  the 
desired  drive,  at  the  main  menu  prompt  enter,   "nsf"      The  command  will  be, 
Onsf     Copy  the  transcript  file  onto  the  hard  disk.     The  search  transcript 
file  was  named  "sss"  where  "sss"  was  'the_number  assigned  to  that  searcher. 
To  copy  the  file  onto  the  hard  disk  use  the  command,  6>copy  a:"sss" 
<This  procedure  is  also  described  in  "P^poeduve  Np,  7"  which  was  an  instracti 
sheet  used  by  project  staff  at  the  time  the  searches  were  being  done.     For  a 
copy  of  ••Procedure  No.  7"  refer  to  the  •^'Procedures"  which  are  the  written 
instructions  used  for  all  the  various  tasks  involved  in  the  project. > 


2.  Make  two  backup  copies  of  tlFie  search  transcript  on  two  separate  diskettes 
T^???_'^*^°  ^??^®ttes  are  labeled  with  the  searcher' s  number  ''sss".  _Project 
searches  were  backed  up  in  a  slightly  different  way.     Because  the  project 
searchers  reused  their  project  searcher  numbers^  e.g. ^  12dj  220,   32dj  420  wer 
used  for  each  Question  project  search  020  did j  the  backup  files  for  the  proje 
searches  carried  batch  designations  in  the  form  of  an  alpha  character.  For 
exampie,   project  searcher  transcript  fi les  included  120-420^  120A-A2bA, 
120B-42QB^  etc.    CThe  corresponding  t^  as  will   be  explained  later  in 
these  procedures^   also  had  a  similar  batch  designation,   such  as,  Si2bA.TXT, 
S220A.TXT,   S32dA.TXT,   SA2dA.TXT,   S120B.TXT,  etc.) 

3.  Type  the  "sss"  f i le  onto  the  screen.  Ose  the  command ,  S>typ3  "sss" 
By  stopping  and  starting  the  scrpi  ling ^  ctri  s^  rnake  sure  a  1 1_ questions 
assigned  to  that  searcher  are  present.     it  is  possible  that  the  searcher 

did  riot  record  all  of  his  or  her  search  on  the  diskette.  _This  can  happen  aft< 
a  cut-off  from  Telenet etc.   _Mhen  a  potion  of  a  search  was  'nissing  from  the 

diskette  but  was  recorded  on  the  print-out ,   the  search  was,   in  most  cases  

rekeyed  by  project  staff.  Alternately ,  in  some  cases^  and  always  if  the  seari 
was  not  on  the  print-out  either,   the  question  was  assigned  to  another  searchei 
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4m     Run  StEI=*6i     this  program  selects  pdrtions  of  the  transcript  desired  for 
forther  study.     the  portions  identified  for  further  study  were     1.  All 
*  searcher  oommandSi     Search  commands  always  follow  a  Dialog  system  prompt.  The 
prompt  in  the  Dialog  system  is  a  2.     All   Dialog  system  resjDdnses  to  a 

searoher  "type"  command  excluding  all  but  the  Dialog  accession  number  of  a 
citation  "typed"     3i     the  cost  and  time  spent  system  response  to  a  *'begin** 

dbmmand  or  a  "logoff"  command.   

STEP6  is  run  by  giving  the  following  command*  S>StEP6 

The  system  will   respond  by  asking  for  the  seaarcher  number.  the 

operator  supplies  the  searche^r's  assigned  number f  "sss". 

When  the  program  is  finished  running ,   look  at  the  file  created  on  the  screen  t 
make  sure  the  program  aocut^ately  selected  the  desired  data  from  the  Dialog 
transcript.     The  file  created  by  STEP6  is  called  the  text  file  and  it  is  named 
S"sss".TXT  where  "sss'V  is  the  number  assigned  to  the  searcher.     the  command  to 
use  to  see  the  text  file  is,     G>type  S**sss" .  TXT 

Especially,  when  viewing  the  text  filie,  make  sure  all  accession  number  sets 
are  present.     If  there  was  a   

typo  in  the  Dialog   "type"  command,  the  program  STEP6  may  have  dropped  the 
corresponding  set  of  accession  numbers.     If  this  happens,  the  "sss*"  file 
must  be  edited  using  PCWrite.     Then  rerun  STEP6. 


INSTRUCTION  FOR  EDITING  USING  PC-WRITE 

PC-Write  was  loaded  into  another  subdirectory , in  this  case,  F>.     At  the  F> 
prompt,  use  the  PC-Write  edit  command:   "ed"  and  name  the  drive  specif ication 
and  file  to  be  edited,   in  this  case  g;"sss".     The  command  would  be  as 
f ol lows: 

F>ed  g: "sss" 

After  editing  the  file,   save  the  file  using  the  PC  Write  procedures  Fl  F3. 
Exit  to  the  system,  Fl  F2. 
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5.     Print— out  the  text  file  using  a  line  print  command,   (l>ipr±nt  S"sss"itxt 

6i.     Compare  the  print -out  of  the  text  file  with  the  original  print-but  of  the 

search  transcript .     Make  sure  all  commands,   ^i^^^s  and  costs^  and  accession 

numbers  retrieved  appear  in  the  text  file.   _Tb?_^®^^  edited  using  PC 
Write  in  the  same  way  as  just  described^     Correct  all  typos.     The  text  file 
must  be  in  a  specified  order  as  described  belows 

?b"bbb"  (first  question)   begin  command  to  enter  the  assigned 

database 

7*q"qqq"  (first  question)  question  number 


the  search  (first  question)including  all   searcher  cornmand  

statements^   and  ail   Dialog  system 
responses  in  the  form  of  accession 
numbers 

?b"bbb"  (second  question)   begin  command  and  database  number 

^  (first  question)   Dialog  response  of  cost  and  time 

online  in  first  database  used 


?*q"qqq"  (second  question)  question  number 

the  search  •  (second  question) 

etc. 

the  search  ( 1 ast  question ) 
?logof f 

^  ( last  question)   cost  and  time 
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7.     Delete  the_"sss"  file  from  the  hard  disk  using  the  command, 
G>del   "ss©"       If  there  were  oorredtidns  necessary  on  the 

"66s"  file  in  order  for  STEP6  to  work  properly,   before  deleting,   copy  the  

pprrected  '*bb^*'  file  over  the  two  copies  of  the  "sss"  files  on  the  two  backup 
diskettes  already  created  in  the  second  step  of  these  instructions. 

6.     Next  run  the  program  STEP7-     This  program  separates  the  text  file 
into  three  categories  of  files: 

ii     Accession  numbers     2i     Commands     3^     Cost  and  timei 

there  will  be  one  of  each  of  these  files  for  earch  question  represented  in  the 
text  file^  for  a  total  number  equal  to  3  times  the  number  of  questions  searohi 
To  run  the  program^  at  the  S>  ask  for  ask  for  the 

searcher  number^     The  operator  supplies  the  searcher  number^  u^^u. 
The  program  runs  and  when  it  is  finished  responds  that  the  original  file  has 
been  separated  into  three  categories  of   fiies^   as  ^ust  described i 
These  files  are  named  S"qqqb  "sss" 
where  ''qqqbbb"  represents  the  question  number/database  number  of  each*  questibi 
searched  and  "sss"  E?PE®^?D^?_^'^?_??^F9^®F1?  D^'^^^Fl     STEP?  creates  a  total  o; 
3  times  the  number  of  questions  searched  for  each  text  file  inputs     After  STEI 
is  run,   the  operator  should  check  that  all  the  desired  S"qqqbbb''i'*sss^^ 
C"qqqbbb"."ss3"  and  T"qqqbbb'' .  "sss"  were  created.     To  do  this^  use  a  director; 
ocimincind  in  the  form,  G>dir  ?"qqqbbb".*       If  they  were  not  ail  created,  there 
is^  a  problem  in  the  tr?xt  file.     Go  back  and  edit  the  text  file  again i  Pay 
particular  attention  to  the  format  of  the  begin  command  and  the  ?*q"qqq" 
question  number  line.     Then  rerun  STEP7. 

Aiter  3TEP7  is  finished,  copy  the  text  file  onto  the  two  searcher  backup 
c  .     :^lfi,t>e^..     Use  the  ccmmand  ,  <3>oopy  S"sss"  iTXT  a :  

Prt7^^'=^^f  ^  searches  are  backed  up  in  batches  on  two  backup  diskettesi  in 
acic . I >;ian,  different  batches  are  given  various  designations  so  as  to  be  able 
to    -Gi  ;  t^hem  apart.     Far  example,   project  searcher  £320  had  the  foi  lowing  tex^- 
flli^G^   il20.TXT,   S220.TXT,   S320.TXT,   SA20.TXT,   Si2bA.TXT,   S220A.TXT,   S320A,T:  ' 
S42C5/..  .  V  Xr,   Si2nB.TXT,  etc. 

Also  owpy  all  01  the  S"qqqbbb" .  "sss"  C"cjqqbbb** .  "sss"^*  and  T"qqqbbb*'."sss"  files 
don^>  by  vvhe  searcher  onto  one  of  the  backup  searcher  diskettes  by  using  the 
commcind  ^ 

G>oopy  ♦.•*sss"  a: 

Aiso  copy  the  T"qqqbbb". "sss"  online  time  files  onto  a  cumulated  diskette  of 
T  Conline>   files.     These  files  have  the  label  TOFiiel  through  TOFiieAi  Th^y 
are  arranged  by  question  numbers.     Questions  1  through  10  are  on  the  TO 
diskette;  questions  11  -  20  on  the  TOFi le2  diskette;  questions  21  -  30  on  the 
TOFileSS  diskette  and  questions  31  -  40  on  the  T0File4  diskette. 
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tOm     Delete  -the  -text  file  from  the  hard  disk  using  the  command, 
g>del  S"BS&''iTXT 

11 i     After  all   searches  of  a  question  are  completed  by  all  B  searchers  and  4 
project  eearchesj   and  a^l  the  T  files  for  the  question  are  created, 

print-out  all  the  S  files  for  the  question ^     there  wiH   be  9  S  files,  5  for  th 
eearohers  and  4  for  the  project  searches.     Use  the  automatic  print  feature  of 
the  DdS^   Ctrl/Prt  Sc^   to  turn  on  the  printer  and  then  use  a  "type"  commands 
and  name  e^ch  desired  file.     the  commands  will  be, 
e>type  S"qqqbbb"."sss" 

12^     Compare  the  print-puts  of  the  S"qqqbbb" s "sss"  files  for  question  "qqqbbb" 
with  the  print—outs  of  the  text  files  for  the  searchers  who  searched  that 
question.     Determine  which  accession  numbers  the  searcher  intended  for  his  or 
her  answers     Searchers  on  Dialog  typed  as  many  citations  as  needed  to  review 
the  results  of  their  retrieval  efforts^     However^  they  were  instructed  to 
designate  those  citations  desired  for  their  final  answers  by  typing  them  in 
Dialog  format  1. 

i3i     Edit  the  S"qqqbbb" i "sss"  files  leaving  only  the  accession  numbers 

that  the  searcher  wanted  as  his  or  her  final   answeri     this  can  be  done 

using  PC  Write  and  following  the  instructions  outlined  above. 

in  some  cases^   the  searcher  will  not  have  produced  an  answer.     If  the 

searcher^'s  result  is  an  empty  set^   replace  the  void  wi'?^h  a  zero,  0. 

in  essence  then^   their  set  will   include  one  accession  number,   a  zero. 

in  the  final  data  files^  QUES"qqq".OUt  and  SEAR"sss" .OUt ,   produced  by 

the  programs  StEPlQ  and  StEPl^     this  result  will   be  represented  as  one 

"not  evaluated"  citation  retrieved  eventhough  they  really  did  not  retrieve 

anything^     This  was  corrected  manually  in  the  final  data  files.     the  following 

searchers  produced  empty  sets: 

Se>archer  D06  Question  607;  SQQS  Q018;  S014  Q018;  S026  Q018;  S025  0024; 
SC3'b  Q02S;   SOOl   Qd33;   3015  Q039 
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lAi     Run  STEPl  and  STEP2. 

STEPl  asks,   "is  this  the  first  time  to  run  this  program?"     Answerj  NO 
<unies5  yoa  are  beginning  with  a  whole  new  project  and  a  new  batch  of  questioi 
STEPl  also  asks  for  the  number  of  question/database  numbers  being  added. 
Answer  as  'nany  as  you  have.     then,  the  program  will  ask  for  specification  of 
the  actual   "qqqbbb"  numbers* 

STEP2  asks  the  operator  first  for  the  number  of  question/database  numbers  to  i 
processed,   then  for  the  specific  "qqqbbb"  number  and  then  for  the  searcher 
numbers  '^sss"  of  the  searchers  who  did  the  question.     the  operator  inputs. 
In  turn,  each  searcher  v^^c  did  the  question.   There  will  be  9  inputs, 
5  for  the  searchers   ■  •.  r  the  project  searohesi 

The  searcher  is  inpi  ,iQ  the  searcher  number  ''sss''i     the  searcher 

numbers  should  be  in:  t£»oendiy?g  order  with  the  pro ject  se-^rch  numbers 

last  and  also  in  &sc?i  :>r.deri     P'or  examplej   if  searchers  001  ^  002^^  003,  OC 

and  005  and  proje=^       c  118  dta  the  question^  the  searcher  numbers 

wc3uld  be  input  one  at  i     x^re     QOi^  then  0C12^  th'rn  003|  then  004  then  005, 
then  the  project  search  numbers  115^   Site?,,   318^   and  finally  A18i 

STEP2  produces  a  Q"qqqbbb" i  *'sss"  file  for  ^ach  searcher  for  the  question 
"qqqbbb"  and  one  Q"qqqbbb"i  DAT  file  for  the  questioni     the  Q"qqqbbb''i  "sss" 
files  are  comparable  to  the  S"qqqbbb'' i  "sss**  files  except  the  accession 
numbers  are  in  orders   _the  Q^'qqqbbb".DAT  file  is  ^_'T'erged  setj^  in  descending 
order ^  of  all  accession  numbers  retrieved  by  all  searchers  for  the  question. 

15 i     Print— out  the  Q"qqqbbb" ^ DAT  file  for  the  question  using  a  line  print.  Tf 

comrriand  should  be^ 

6>1 print  Q"qq§bbb" . DAT 

Doing  a  line  print  of  the  Q"qqqbbb"^DAt  file  tells  you  how  many  accession 
numbers  are  included  in  the  merged  set. 

16i     ^opy  the  edited  S"qqqbbb'' i  "sss"  files  onto  backup  diskettes  of  cumulated 
edited  S"qqqbbB"  i  "bss*^"  filesi     These  files  are  labeled  SFilel  through  SFileA 
according  to  the  questions  included  on  each  diskette.     SFilel  includes 
questions  1   -  10;   SFile2  questions  11  -  20;   SFile3  questions  21-30;  and 
SFile4  questions  31  -  AOs 

Delete  the  edited  S"qqqbbb" . "sss"  files  for  the  question  using  the  command, 
S>del  S-^'qqqbbb".* 
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17.  Run  STEPS  it        lis  program  prepares  the  merged  accesBion  number  set  &o  it 
can  be  uploaded  to  Dialog  and  full   records  with  abstracts  downloaded.  The 
program  does  this  by  adding  a  Dialog  "keep"  command  to  each  accession  number 
in  the  file  and  creating  a  new  file  named  the  Q"qqqbbb" . COD  file. 

The  program  asks  for  the  "qqqbbb"  number.     The  operator  supplies  it. 

18.  Copy  the  Q"qqqbbb" . COD  file  onto  a  new  diskette.     Use  the  command, 
6>cbpy  Q"qqqbbb".COD  a: 

Delete  the  Q"qqqbbb" . COD  file  from  the  hard  disk  using  the  command, 
e>del  (S"qqqbbb".COD 

19.  Upload  the  Q"qciqbbb". COD  file  from  the  diskette  to  Dialog  using  1200  baud 
Smartcom.     Download  the  first  150  abstracts  onto  the  diskette.     Name  the 
downloaded  abstract  file  Q"qqqbbb" . ABS 
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DiAtOG  UPLOADINS/DOWNLOADING 

bogon  to  Dialog2  using  the  12DD  baud  Smartcpm  mbuhted  dh  the  pc.     From  the 
Smartc5ofn  menu  ask  for  a  "send  file"  option  3,   using   "send   lines"  mode 
option  5-     The  name  of  the  file  to  send  is  the  Q"qqqbbb" .  COD  file.     This  file 
is  on  a  floppy  diskette  that  has  been  loaded  into  the  auxi 1 iary  drive. 
The  Smartcom  software  asks  for  namepf   file  to  send.     The  operator  responds 
with  the  drive. designation  and  the  file  name.     The  command  will   look   like  this 
NAME  OF  TD  SEND  FILE:   a  :Q"qqqbbb»' .  COD       At  this  point,   the  software  will  begin 
sending  the  file  to  Dialog  and  Dialog  will   respond  by  creating  a  set  including 
#ll_pf  the  accession  numbers  seht._  When  the  entire  s  the 
software  will   respond  with  a  message  "Send  complete".     The  operator  will  key  i 
Fl  Using  the  Fl  key.     Then  from  the  Smartcom  menu  the  operator  should  select 
"receive  file"  option  4,   and  "stop/start"  option  2.     The  software  will   ask  for 
the  name  of  the  file  to  receive.     From  the  Dialog  system  three  files  in  total 
will  be  received.     1.     all  accession  numbers  sent     2.     accession  number  1-150 
3.     abstracts.of   records  1-15D.     In  cases  where  the  merged  set  of  accession 
numbers  is  150  or  less,   receive  only     1.     all   accession  numbers  and     3.  all 
abstracts-       Set  1   is  named  Q"qqqbbb" .ACCj  set  2  is  named  Q"qqqbbb" . AC; 
and  set  3  is  named  Q"qqqbbb" . ABS.     Each  of  these  files  should 
be  received  on  the  diskette  in  the  auxiliary  drive-.   Therefore  each  file 
should  be  namied  with  an  a:  drive  designation  as  follows: 

asQ"qqqbbb".ACC;       a :Q"qqqbbb" . AC;  a:Q"qqqbbb".ABS 

After  naming  the  file  to  receive,   the  software  sends  the  operator  back  online. 
The  operator  enters  the_ appropriate  Dialog   ''type"  command  for  the  desired 
Dialog  system  output.     For  the  accession  number  set  the  Dialog  command  is, 
?tO/l/all 

For  the  150  accession  number  set,   the  Dialog  command  is, 

?tD/l/l-"150 
For  the  abstract  set  the  Dialog  command  is, 

?tO/5/l-150   Cif    less  than  15D  in  the  set  use  ?tO/5/all) 

After  downloading  each  file  hit  the  Fl   key  to  close  the  file.     After  ail  three 
are  downloaded,   logoff   Dialog.     Then  end  the  Smartcom  software  using  "end 
communication"  option  O. 
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20.  Copy  the  abstract  file  from  the  diskette  into  the  hard  disk.  The  commah 
fep  use  is J 

G>copy  a:Q*'qqqbbb". ABS 

21.  _    Run  STEP5.     This_ program  inserts  a  rereyahce  judgement  blank  at  the  end 
each  abstra.'Dt  in  the  Q"qqqbbb" •  ABS  file._  After  asking  for  STEP5  the  system 
asks  for  the  question/database  number .     The  operator  supplies  tbe_"qqqbbb" 
number.     The  abstractf ile  with  evaluation  blanks*   produced  by  STEPS*  has 
the  name  F'^qqgbbb"  . ABS 

Type  put  the  F"qqqbbb" . ABS  file  on  the  screen  using  Otype  F"qqqbbb" • ABS 
to  make  sure  that  the  evaluatibn__blanks  are  in  correct  position  at  the  end 
of  each  abstract.     The  program  STEP5  may  insert  extra  blanks  if  there  are 
slashes  appearing  in  the  text  of  the  abstracts.     This  is  common  in  abstracts 
about  chemistry.     The  F"qqqbbb" . ABS  file  can,   if  necessary*   be  edited  using 
PC  Write  and  the  procedures  for  using  PC  Write  outlined  above. 


22.  Delete  the  Q"qqqbbb" . ABS  file  iroT*  the  hard  disk  using  the  command, 
G>del  Q"qqqbbb".ABS 


23.     Print  the  F"qqgbbb".ABS  file  onto  2-part  form  using  the  command* 
G>print  F"cjqqbbb" .  ABS 

2^.  Delete  the  F"qqqbbb" .  ABS  file  frejn  the  hard  disk  using  the  command* 
G>del  F"qqqbbb".ABS 

25.     Send  the  abstract  set  and  a  copy  of  Form  12  to  the  user. 
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IV.B.   PROCEDURES  FOR  PROCESSING  EVALUATED  ABSTRACTS  AND  FOR  PRObOClNG 
FINAL  DATA  SUMMARIES 


The  process  described  in  the  following  steps  is  done  after  the 
evaluated  set  of  abstracts  is  returned  by  the  user.  The  procedure  for 
processing  the  evaluated  abstracts  and  producing  final  data  summaries  follows 


1.     The  data  files  and  eKeciitibn  programs  required  to  process  the  data  at  thi! 
point  are  all   located  in  the  same  directory  as  the  programs  and  files  describf 
in  the  first  part  of  the  step-by-step  procedures  in  this  report.     On  the  IBM/J 
the  programs  and  files  are  accessed  using  the  command,  nsf ,   *rbm  the  first 
prompt  after  the  main  menu  is  displayed.     After  this  opening  command,  the 
operator  will  have  a  G>  prompt. 

2-  With  the  evaluated-  abstract  se  in  hand,  the  operator  asks  for  the  progran 
STEP4  using  the  command,  G>STEP4 

The  program  responds  by  asking  for  a  question/database  number.     The  operator 
supplies  it.     Next  the  program  will  ask  how  many  references  will  be  reviewed. 
The  operatDr  responds  with  the  exact  number  of  abstracts  in  the  set  sent  to  th 
user.     Next  the  program  will   begin  prompting  for  each  accession  number  of  each 
^^stract  in  the  set.     These  prompts  should  follow  the  order  of  the  accession 
numbers  of  the  abstraDts  in  the  printed  abstract  set.     As  each  accession 
number  is  displayed,   the  operator  supplies  the  relevance  judgement  provided  b> 
the  evaluator.     The  operator  will  enter  either 

Rf   Pf         or  E  where  R  -  Relevant,   P  -  Partially  Relevant,   N  -  Not 
Ffelevant,   and  E  -  Not  Evaluated  are  the  codes  used  for  the  evaluation 
judgements.     After  all  of  the  judgements  haye_ been  entered,  the  system 
responds  with  the  message  that  the  F"qqqbbb" . DAT  evaluated  accession  number 
file  has  been  created  and  that  the  relevance  judgements  just  supplied  can _be_ 
printed  out.     The  file  that  can  be  p'^inted  out  is  called  the  Q"qqqbbb"  .OUT  fil 


3.     Print  out  the  Q"qqqbbb". OUT  file  using  the  cbmmahdj   GMprint  Q"qqqbbb"  .OUT 
Compare  this  printed  file  with  the  printed  abstract  set.     Make  sure  all 
relevance  judgements  were  assigned  correctly. 
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4i  Now  the  final   jadgement  assignments  can  be  made.     The  _ 

operator  should  ask  for  the  program  STEP9,   using  the  command  G>STEP9 
The  program  will  ask  and  the  operator  will  supply  questibh/database  number. 
The  program  will  then  ask  if  the  operator  wants  to  edit  or  make  fin^l   form.  .1 
errors  were  found  in  the  printed  Q"qqqbbb" .OUT  file,   these  can  now  Le  correcti 
by  asking  to  editi     The  operator  asks  to  edit  by  answering  e  to  the  following 
question 9 

Do  you  want  to  edit  or  make  final  form  (ie/f)?     e  _ 
The  system  will  ask  for  the  accession  number  to  be  edited.     The  bperatbr 
supplies  the  accession  number.     The  system  displays  the  current  judgement.  Th 
operator  can  input  a  new  valt«e.     This  process  continues  until  the  operator 
answers  no  to  the  question.  More  editing?  n 

At  the  beginnifig  of  STEP9,   if  the  operator  doesr  not  want  to  edit, 

the  first  question  can  be  answered  f^ 

bo  you  want  tci.  edit  or  make  final   form  <e/f>?  f 

the  system  will,  at  both  the  end  of  editing  IbojD  and  at  the  "f",   final  fbrm 
command^  produce  the  final  evaluated  accession  number  sets  for  all  accession 
number  sets  for  that  question.     This  includes  the  F"qqqbbb" . DAT  merged  set 
and  the  9  F ''qqqbbb'' .  "sss"  sets  for  the  9  individual   searches  of  the  question. 

Sm     Copy  the  Q"qqqbbb" . OUT  file  onto  the  final  data  diskette  for  the  question. 
These  diskettes  are  labeled  Q''qqqbbb"i     There  is  one  diskette  for  each 
question.     Delete  the  Q''qqqbbb"  iOtJT  file  from  the  hard  disk. 

hi     eopy  the  Q"qqqbbb"iLST  file,   the  Q"qqqbbb" .  DAT  file,   the  F"qqcibbb'* .  DAT 
file^   and  the  9  Q"qqqbbb"  *  "sss"  files  and  the  9  F"qqc|bbb" .  "sss"  files  to  the 
final  data  diskette  for  the  question  labeled  Q**qqqbbb**       Alsb,   copy  the 
Q"qqqbbb"iDAT  file  and  the  9  Q"qqqbbb". "sss"  files  to  thecumulated  Q  Files 
diskettes.     These  diskettes  are  labeled^   QFilel  thrbugh  QFileS  according  to 
the  question  number  included  on  the  diskette^     QFilel   includes  questions  1-5? 
QFile2  questions  6-10;   QFileS  questions  11-15 ;   QFi le4  questions  16-205  QFile5 
questions  21-25;   QFile6  questions  2S-30;   QFile7  questions  31-35;  QFileS 

questions  36-4D.  _  _   

Also  copy  the  F"qqqbbb" . DAT  file  and  the  9  F"qqqbbb" . "sss"  f i les  to  the 
cumulated  F  File  diskettes.     These  diskettes  are  labeled,   FFilel  through  FFile 
according  to  the  question  numberr>  included  bh  each.     FFilel   includes  questions 
1-10;   FFile2  que^stions  11-20;   FFile3  questions  21-30;   FFileA  questions  31-AO. 
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7i'     Delete  the  Q"qqqbbb'' .  DAT  file  and  the  9  Q^'qqqbbb'*  * '^sss"  files  from  the 
harddisk.     This  can  be  done  in  three  commands^   G>del  Q"qqqbbb" . DAT , 
G>del  Q"qqqbbb".0??,   G>del   Q"qqqbbb" . ?20  (or  ?19> 


S._  Next  the  pperatbr  should  complete  the  time  files.     STEPS  is  used  to  add  th 
offline  time  to  the  bhlihe  time.     The  online  time  is  already  p-esent  in  the 
T"qqqbbb*V.  "sss**  file.     The  pperatbr  will   supply  an  offline  tine  and  the  progra 
will   add  the  two  times  together. 

Before  beginning  STEPS  the  bperatpr  should  check  the  leriG*,h  c/f  each 
T"  iqbbb" . '"sss"  file  for  the  questipn.     To  dp  this_use_«  T^/'ectory  command, 
G>dir  T**qqqbbb",»     The  onlju^e  time  file  should  be  26,   27,   28  or  29  bytes  in 
length.     If  the  file  is  longer  as  it  will   be  if  Dialogl  was  used,  the 
T"qqqbbb** . "sss"  f ile  must  first  be  edited  using  PC-Write   (procedures  outlined 
in  part  one  of  these  Step-by-steps)   to  eliminate  words_such  as  "9Descri^;bors" 
Then  the  operator  asks  for  STEPS  using  the  command,  G>STEP8 

The  program  asks  for  the  question/database  num  operator  supplxe^  iw. 

The  system  will  access  the  Q"qqqbbb" . LST  filefpr  the  question  anJ*  using 
the  information  in  that  file,   prbmpt  for  ah  of  f  line  time  for  each  searbHe*. 
found  in  the  Q'jqqqbbb** .  LST  file.     The  operator  supplies  the  time  in  rnihutes 
from  the  work  form  used  by  the  searcher  tb  prepare  the  question  offline.  The 
program  converts  the  offline  time  to  a  fraction  pf  an  hpu^  and^.dds  the  value 
to  the  file.     Then  the  program  adds  the  online  time  to  the  offline  time_and 
adds  a  total  t±m   value  to  the  T"qqqbbb" .  "sss"  file.     Aftei^  running  STEPS  for 
the  question,   all   T"qqqbbb" . "sss"  files  should  be  inspected  to  make  sure  the 
program  cpnverted  the  original  T"qqqbbb" .  "sss**  f  i  le  prpperly  .     Thir;  can  be  don 
by  usihg_the  directbry  cpmmahd  and  inspecting  the  length  uf  the  files  produced 
The  hew_T"qqqbbb"."sss"  files  shbuld  be  103  bytes. 

If  any  T'*QQ9bbb" .  ••sss*'  file  was  not  cbnyerted,   CFLAG  program  can  be  used  to 
enable  STEPS  to  run  again.     Make  sure  the  Online  T"qqqbbb" . "sss"  file  is 
prbper.     Ask  fpr  CFLAG  using  the  cbmmand ,  G>CFLAG 

The  prbgramwill   ask  fbr  the  question/database  number .     When  supplied,  the 
program  will  display  in  turn  each  searc^ar  number  and  ask  if  you  want  to  chang 
the  flag  for  that  searcher.     When  the  searcher  number  of  the  T'*qqqbbb'* .  "sss" 
file  that  was  hbt  cbnyerted  is  displayed  answer  yes  to  the  question.   The  flag 
on  searcher  "sss"  is  1  do  you  want  to  change  it  to  O'^  y 
STEPS  can  now  be  rerun. 


?•     Copy  the  Total   Time  T"qqqbbb" . "sss"  files  to  the  final   question  data 
diskette  fbr_the  question.     The  final  questipn  data  diskettes  for  each  questio 
are  labeled  Q"qqqbbb" .     Also  copy  the  Total  Time  T"qqqbbb"^'*sss*'  files  to  the 
cumulated  T  File  diskettes.     These  diskettes  are  labeled  TFilel  thrbugh  TFile^ 
according  tb  the  questipns  included  oh  the  diskette.     TFilel   includes  questioni 
1-10;   TFile2  quesitons  11-205   TFileS  questions  21-30;  TFileA  questions  31-40. 
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10.     Search  therms  used  by  each  searcher  for  each  question  are  determined  by  tl 
program  STEPll  .     The  operator  aasks  for  STEPlli^     The  system  asks  for  t 
appropriate  question/database  number i     The  program  then  processes  the 
C'qqqbbb';.  "sss"  files  arid  creates  a  E)"qqqbbb*' i^'sss**  f  iie  for  eachi     When  the 
program  is  finished  a  descriptor  count  for  each  searcher  is  displayed.  This 
information  is  stored  by  the  program  for  use  iri  final  data  summary  programs. 


11.      Copy  the  9  C'qqqbbb" . "sss"  files  arid  the  9  D"qqqbbb" i "sss"  files  onto  thu 
final  question  data  diskette  labeled  Q''qqqbbb''  and  on  the  cumulated 
CD  File  diskettes.     The  cumulated  CD  File  diskettes  are  labeled  CDFiiel 
through  CDFile?  acc^^rdirig  to  the  questions  included  on  each  diskettes  CDFile 
bbntains  questions  1-6;   CjE)File2  questions  7  -  12;   CDFi le3  questions  13  -  IS 
CPFile4  questions  19  -  24;   CDFileS  questioris  25-  36;   CbFile6  questions 
31-36;  and  CDFile?  questions  37  -  AD. 


12.     Wheri  all  of  the  F"qqqbbb" .  "sss"  files,   the  F'*qqqbbb"  s  DAT  fiii^   ail  bf  th< 
C*'qqqbbb".  •■sss"  and  corresponding  D"qqqbbb"i**sss"  files  snd  all  of  the 
Total  Time  T"qqqbbb" . "sss"  files  have  been  prepared  for  a  given  question,  the 
final  data  summary  about  that  questiori  cari  be  produced ^     Also,  the  final  data 
summ^nry  about  each  searcher  who  did  the  question  oari  be  updated  to  include  thi 
question.      riis  programs  which  produce  the  firial  data  summarier>  are  STEPIQ  and 
STEP12.     Before,  running  either  of  these  programs^   make  sure  that  there  is  a 
QUES"qqci".LST  file  named  for  the  questicari  arid  a  SEAR"sss"  .LST  f  ilG  nailed  for 
each  searcher  who  did  the  question.     These  files  are  riamed  manually  usirig  the 
line  editor. 


13.     STEPID  ban  now  be  run  for  the  question.     Alter  askirig  for  STEPiO,  the 
program  will   ask  for  the  question/database  number       The  program  then 
proceeds  to  access  the  files  just  described  and  updates  the  fiaril  data  summary 
of  each  searcher  who  did  the  question. 


14.     The  f  iri^l  _data_sLimmary  _about  the  questiori  can  also  be  produced.  The 
operator  asks  for  STEP12.     The  program  asks  for  the  question/database  number. 
The  program  then  asks  the  operator  to  supply  the  data  from  Form  12  "User 
Evaluation  of  Answers".     The  program  theri  produces  the  final  data  summary  abou 
the  question  in  a  file  named  QUES"qqq " .OUT       As  described  in  IV. A, 13.   of  thesE 
procedures,  where  a  searcher ' s  empty  accession  number  set  was  replaced  with  a 
zero,  the  final  data  will   be  off   by  1   in  the  "#  not  evaluated"  and  "total 
references  retrieved"   listirigs  for  both  the  question  as  a  whole  and  for  the 
individual   searcher  who  produced  the  empty  set.     These  values  should  be 
corrected.     This  can  be  done  manually,   although  the  programs  could  be  revised 
"to  check  for  a  zero  as  an  accession  number.     The  accession  number  zero  should 
not  be  counted . 
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15.     Print  the  final  question  data  using  the  command,   G>print  QUES"qqq" .OUT 


16i     Copy  the  QUEf''qqq''«bUT  f ile  to  the  final  question  diskette  al ready 
described  haying  the  label  Q"qqqbbb".     Also  copy  the  QUES"qqq" .OUT  to  the 
cumulated  final  question  data  file  labeled  QUESOUT. 


17  i  Delete  the  e"qqqbbb"."sss**  b"qqqbbb" .  "ess"  and  Total  T"qqqbbb" .  "sss"  file 
from  the  hard  disk. 


16 «.  When  all  the  questions  searched  by  a  given  searcher  have  been  processed 
through  STEP12*  the  searcher  final  data  file  is  also  completer  This  file  is 
named  SEAR ''sss^idOT     As  descri^  of  these  procedures^  for 

eearchers  who  Pt^pduced  an  empty  set  of  accession  numbers^   the  empty  set  was 
^^Pi^P®^.**?*^!!  a  zero  as  an  accession  number.     This  zero  was  counted  as  an 
accession  number  by  the  program  STEPld  when  it  should  not  have  beeni     The  fine 
data  tallies  in  the  categories  "#  not  evaluated"  and  "Total  number  retrieved" 
for  the  question  as  a  whole  and  for  the  individual  searcher  producing  the  empt 

set  were  off  by  one  extra  count.     This  was  corrected  manual ly  after  the  

SEAR "sss". OUT  files  were  complete .     The  tal 1 ies  for  the  question  as  a  whole 

were  corrected  in  each  SEAR"sss".6UT  f ile  of  each  searcher  who  did  the  

question.     The  tallies  in  the  individual  searchers  data  was  only  correct «?d  in 
his  or  her  SEAR"sss". OUT  file.     When  the  SEAR"sssi\.OUT  file  for  a  searcher  is 
complete,  copy  the  file  to  the  searcher  backup  diskette  (described  in  part  1  c 
this  step-by-step> .     Also  copy  the  file  to  the  cumulated  search  final  data 
diskettes  labeled  SEAROUTl  and  SEARbUT2.     SEAROOTl  includes  searchers 
001  -  020;  SEARbUT2  includes  searcher^  . 
^^^_f^Dal  ^^archer  report  can  be'  printed  using  the  command, 
e>print  SEAR"ss5".6UT 
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V.  FILES  USED  AND  CREATED  BY  EACH  PROGRAM 


File  Name 


Descript^ion 


"sss"  The  Dialog  transcript  was  created_bhl ine  and  was  processed 

by  STEP6.     There  were  a  total  of  36  of  these  fil&s^   one  for 
each  searcher  who  participated.     Additionally,   there  were 
a  number  of  these  files  for  project  searches. 

S"sss'*.TXT  The  text  file  for  each  searcher  was  created  by  STEPS  and 

processed  by  STEP?.     Again,   there  were  a  total  of  36  of  these 
files.     Additionally,   there  were  a  number  of  thes'v  files  for 
project  searches. 

tOTAt-tSt  The  total   list  file  was  a  master  list  of  all  questions. 

Created   (initial ly>   and  subsequently  updated  by  STEPl . 
There  was  only  one  TOTAL. LST  file. 

Q"qqqbbb''.LST       The  question  list  files  were  master  lists  of  al  1  searchers 

of  each  question.     Named  by  STEPl ,  data  was  input  by  STEP2. 
There  were  a  total  of  40  of  these  files,   one  for  each  question 

S"qqqbbb". "sss"  The  S  files  included  type  commands  and  accession  numbers. 

Created  by  STEf=>7,   edited  using  PC  Write  or  WORDSTAR,  the 
edited  versions  were  processed  by  STEP2.     There  were  9  of  thes 
files  for  each  question,   and  a  grand  total  of  360  for  all  AO 
questions. 

C'qqqbbb".  "sss*'  The  C  fi les  included  al,l  commands  used  by  a  searcher  for  a 

question.     Created  by  STEP7,   they  were  proqessed  by  STEPl 1. 
They  are  used  as  datc:^  input  to  STEPIO  and  STEP12. 
There  were  a  total  02   9  of  these  files  for  each  question,  and 
grand  total  of  360  files  for  all  40  questions. 

T"qqqbbb" . "sss"  The  T   Conl ine)   files  included  time  and^ cost  online. 

Created  by  STEP7,  they  were  processed  by  STEPS. 

STEPS  produced  an  edited  version  of  the  f i les_including  time 

online,  time  offline,   and  total  time  spent.  The  revised 

T"qqqbbb". "sss"  was  used  as  data  input  to  STEPIO  and  SfEP12. 
There  were  a  total  of  9  of  th^se  files  for  each  question,   and  1 
grand  total  of  360  files  for  all  40  questions. 

Q"qqqbbb''.  ••sss"  the  Q  files  were  sorted  accession  numbers  files.     Created  by 

STEP2,   they  were  processed  by  STEP9.     There  were  a  total  of  9 
of  these  files  for  each  question,   and  a  grand  total  r  1  360  for 
aii  40  questions.  ^ 
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v.  Cont. 


Q^qqqbbb" .  DAT       T'^?  Q   iDAT  files  included  merged,  sorted  accession  numbers  for 

each  qaestion^  Created  by  STEP2j  they  were  processed  by  STEPS 

and  STEiPAi     There  were  a  total   of  40  of  these  files,   one  for 
each  of  the  questions. 

Q"qqqbbb".COD       the  Q   ^ebD  files  inciaded  the  merged  accession  number  sets 

with  Dialog  "keep"  commands  -     Created  by  STEP3,  they  are 
processed  by  Dialog.     There  were  a  total   of  40  of  these  files, 
one  for  each  question. 

Q"qqqbbb". ABS       The  Q   .ABS  files  contained  abstracts  downloaded  from  Dialog. 

They  were  processed  by  STEPS-     There  were  a  total  of  40  of 
these  files,   one  for  each  question^ 

F"qqQbbb" . ABS       The  F   .ABS  files  were  sets  of   abstracts  with  evaluation  blanks 

inserted .     Created  by  STEPS ,   they  were  printed « 

F**qqqbbb" .  **sss"  The  F  files  were  accession  numbers  with  evaluations.  Created 

Cbr  edited)   by  STEP9  they  were  used  as  data  input  to  STEPIO 
arid  STEPi2.     There  were  a  total  of  9  of  these  files,   and  a 
grarid  total  of  360  files  for  all  40  of  the  questions. 

F^qqqbbb" . DAT       The  F   .DAT  files  were  merged  accession  numbers  with  evaluation 

for  the  question.     Created  by  STEP4^   edited  by  STEP9  they  were 
used  as  data  input  to  STEPIO  and  STEP12.     There  were  a  total 
of  40  of  these  files,   brie  for  each  question. 

Q"qqqbbb" .OUT       The  Q   .OUT  files  were  merged  accessiori  numbers  with  evaluation 

in  a  reada b 1 e  f brmat .  Created  by  STEP4 ,  they  were  ed i ted  by 
STEP9  and  printed ^  There  were  a  total  of  40  of  these  files, 
one  for  each  questiori. 

D^qqqbbb" . "sss"  The  D  files  were  lists  of  search  terms  used  by  each  searcher 

for  each  question.     Created  by  STEPll,  they  are  used  as  data 
input  to  STEPIO  arid  STEP12.     There  werJ?  a  total   of  9  of  these 
files  for  each  question,  and  a  total  of  360  files  for  all  40 
questions. 
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SEAR"sss".LST       This  file  is  used  by  the  prc3gram  STEPIO  in  its  operation.  The 

SEAR  .LST  file  is  named  manual ly  using  the  line  editor.  There 
must  be  one  of  these  files  named  for  each  searcher. 


QUES"qqq".LST       This  file  is  used  by  the  program  STEP12  in  its  operation.  The 

QUES  .LST  file  is  named  manual ly  using  the  line  editor.  there 
must  be  one  of  these  files  named  for  each  question. 


SEAR"Sss" .OUT       The  firt'^l  searcher  data  file  was  created  and  subsequently 

updated  by  STEPIO.     There  were  a  total   of  36  of  these  files, 
one  for  each  searcher  who  participated .  Additionally, 
there  were  12  of  these  files  for  project  searches. 


QUES"qqq".OUT       The  final  question  data  file  v/as  created  by  STEP12. 

There  were  a  total  of  40  of  these  files,   one  ft^r  each  question 
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VI i  FILES  GATEGORIZED  BY  FILE  CONTENT 


Transcript  File 
••bss" 


Text  File 

S"sss"-TXt 


List  Files 

TOTAL. LST 
Q'*qqqbbb".LST 


QOES"qqq".L&T 
SEAR*'sss".LST 


Accession  Numbers 

S"qqqbbb"."sss" 

S"qqqbbb"."sss" 

Q"qqqBbb".''sss*' 
F"qqqbbb"- "sss" 


Q"qqqbbb"-DAT 
F"qqqbbb".DAT 

Q"qqqbbb"-OUT 

Q"qqqbbb"-COD 


Abstracts 


Command! 


Q"qqqbbb".ABS 
F"qqqbbb*'.ABS 


C**qqqbbb"."sss" 
b"qqqbbb"."sss" 


Exact  transcription  cf  search  session,  ASCII  text  file 
Created  onl ine  to  Dialog . 


Edited  transcript  to  select  command  lines ^  accession 
numbers f   and  cost  and  time.     Created  by  STEP7i 


Master  question/database  iisti     Created/updated  by 
STEPi.  

List  of  searchers  of  each  question i     Named  by  STEPI. 

Data  filled  in  by  STEP2^ 
Used  for  program  STEPi2  operation.     Named  manually. 
Used  for  program  STEPlQ  operation.     Named  manually. 


Unedited   ^includes  type  commands^   for  question  by 
searcher  m     Created  by  STEP?  i  * 

Edited^includes  only  final  set  or  sets)   for  question 
by  searchers     Created  manual ly^ 

Sorted  for  question  by  searchers     Created  by  STEP2. 

Sorted  with  evaluations  for  question  by  searcher. 
Created  by  STEP9. 

Merged  for  question i     Created  by  STEP2. 
Merged  with  evaluations  for  question.     Created  by 
STEP4i 

Merged  with  evaluations  in  readable  format  for 
question  i     Created  by  STEP4 .  _ 

Merged  with  Dialog  keep  cammands  for  question.  Creates 
by  STEPS. 


bownioaded^     Created  online  to  Dialog.  _ 
With  evaluation  blanks.     Created  by  STEPS. 


Ail  command  lines  for  question  by  searcher.     Created  bj 
STEP7. 

Search  terms  for  question  by  searcher.     Created  by 
STEPI i . 
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Tine 


T"qqqbbb" -  "sss"  Online  trime  for  ques-tion  by  searcher.     Created  by 

STEP7. 

T"qqqbbb" •  "sss"  To-bal  -time  (revised  version  pf  original 

T"qqqbbb" .  "sss")  for  quest'ion  by  searcher.  Created 
by  STEPS. 


Searcher  Data 


SEAR"sss".OUT       Final  data  summary  about  each  searcher.     Created  by 

STEPIO. 


Question  Data 


QUES"qqq".OUT       Final  data  summary  about  each  question.     Created  by 

STEP12. 
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VII.  FILE  STORAGE 


Searcher 
Backup  1 
SOOl-SOAl 

File 
Nal«e«»* 


"sss" 

S"SBB".TXt 

Unedited 
S 


Online 
T 

Edited 
S 

total 
T 

Q,  Q.DAT, 
Q.LST 


F,  F.DAT 

Q.OUT 

CHJES"qqq". 

out 

SEAR "sss". 

OUT 

SEAR  it ST 
QiJES  .tST 


X 

X 

X 
X 


BAt-^up  Files<^^ 

i>iskette/L-ii;-^  / 
^irial 


Searcher 
Backup  2 
S001A--S041A 


X 
X 


Question 

QD01"bbb"- 

QOAO"bbb" 


X 
X 
X 


Cumulated 
by  File 
Type»* 


X 
X 

with  C 
X 


X 
X 
X 


Delete 

from 

G> 


after  STEP? 
after  STEP? 

N/A 

after  STEP12 
N/A 

after  STEP2 

after  stEP12 

after  stEP42 
except  Q.LST 

after  STEP12 

Do  Not  Delete 

after  STEP4 

Do  Not  Delete 

Do  Not  Delete 
Do  Not  Delete 
Do  Not  Delete 


«There  are  also  AO  diskettes  labeled  QOOl "bbb"A-Q040"bbb"A  which  have  the  Q.CO 

and  QiABS  files  for  each  question 
»»For  volume  labels  of  cumulated  files  see  next  page 

«»»File»  named  Unedited  S,   C,  Online  T,  Edited  S,  Total  t,  Q,  D,  and  F  each 
include  "qqqbbb" . "sss"; 
Files  flamed  Q. DAT,  Q.LST,  Q^COD,  Q.ABS,  F.DAT  and  Q.OUT  each  include 
"qqqbbb"  before  the  extension 
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Vlli  Cont. 

Files  Cumulated  By  File  type 


Volume  Files   Questions 

Labels  Included  included 

SFilei  edited  1  -ID 

2  S"qqqbbb". "sss"  11-20 

3  21-30 

4  31-40 


tOFilel  Online  time  1  -10 

2  t"qqqbbb"i "sss"  11-20 

3  21-30 

4  31-40 

TFilei  Total  time    1-10 

2  T"qqqbbb".;"sss"  11-20 

3  •  21-30 

4  31-40 


QFilei  i.   Q"qqqbbb" ^ "sss"  1  -5 

2  2.   Q"qqqbbb"ibSt  6  -10 

3  .  3.  Q"qqqbbb".bAt  11-15 

4  16-20 

5  21-25 

6  26-30 

7  31-35 

8  36-40 

CDFilel  .        ii   e"qqqbbb" . "sss"  1  -6 

2  2.   b"qqqbbb". "sss"  7-12 

3  13-18 

4  19-24 

5  25-30 

6  31 -36 

7  37-40 


FFilei  i.  F"qqqbbb" . "sss"  1  -10 

2  2.   F"qqqbbb".E)AT  11-20 

3  21-30 

4  31-40 

QUESOUt  QUES"qqq".OUT  1  -40 


SEAROUtl  SEAR"sss".OUT  Searcher  1  -20 

2  Searcher  21-41 

(.not  22  or  31) 

SEARLSt  SEAR "sss ".LST  all  searchers 

QUES"qqq".LST  all  questions 

StXt  S"sss".TXT  Searchers  1  -41 
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VJIIi  EXAMPbES  OF  FIbES 

The  foi lowing  pages  show  exaoiples  of  the  various  files  wsetS  and  craated  by  th 
programs  described  in  this  repbrti     The  name  of  each  file  appear:^  in  capital 
letters  followed  by  the  file  contents.     The  following  •;<afnples  w&x>e  taken  fro 
question  029;  searcher  039. 

A.     8039.TXT     <Only  one  question  included > 


?bS 

?«cie29 

?s  microwaves 

?s  ceramic  materials 

?s  ceramic  materials  -  sintering 

7c  i  and  2 

7o  1  and  3 

?c  A  or  5 

7s  ceramic? 

?s  fir? 

?s  firing 

?B  sinter? 

?s  microwave? 

?s  radiation 


?c  11 
?ds 
?c  7 


and  12 

and  9 
and  10 
and  11 
and  13 


?c 
?c 
?c 

?dt- 

?c        and  <9  or  103 
?t  o/3/l-lO 
1492491  1366093 


and  13 


1281300 
0720468 
and  11 


1130844 
03081 80 


0740589  0721339 
?c  14  and   (9  or  103 
?t  19/3/1-S 
1436463  1368412 
?c  6  or  19 
?t  20/1^1-15 
1436463  1402491 

740589  721339 

294930 
?b  5 

«4.02    0.268  Hrs  File8  10  Descriptors 


0506806 


1368412 
720468 


0383544 


1366093 
506806 


10S950B 
0294930 


0341824 


12S1300 
383544 


1130844 
341824 


10S950S 
30S180 


5B2 

ERIC 


Procedure  12  -  p.  34 


VIII.  Coht. 


B.     8029008.039  (Unedited) 


t  6/3/i-id 

14i32491  1366093         1  261300 

Q74QS89  0721339  0720468 
t  19/3/1-5 

1436463  1368412  0506806 
t  20/1/1-15 

1436463  1402491  1368412 

740589  721339  720468 

294930 


1130844 
0308180 

0383544 

1366093 
506806 


1089508 
0294930 

0341824 


1281300 
383544 


1130844 
341824 


1089508 
308180 


Ci     S029O08.039  (Edited) 

1436463  1402491  1368412  1366093  1281300  1130844  1089508 
740589  721339  720468  506806  383544  341824  308180 
294930 


Di  C029O0S.039 
5  microwaves 

s  ceramic  materials  . 

s  ceramic  materials  —  sintering 

c  1  and  2 

c  1  and  3 

o  4  or  5 

s  ceramic? 

s  fir?  

s  firing 
s  sinter? 
s  microwave? 
5  radiation 
c  11  and  12 
ds 

o  7  and  9 
c  7  and  10 
c  7  and  11 
c  7  and  l3 
ds 

o  14  and   <9  or  10)   and  13 
t  6/3/1-10 

C3  14  and   (9  or  10)  and/ 11 
t  19/3/1-5  / 

o  6  or  19  /  tort 

'^0/1 /I -IS  /  Oh  3 
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VIII.  Cont. 

E.  f029008.039  (Oniine) 
«4.02     0.268  Hrs  FileS 


F.     T029008.039  <Tofcoal) 

p. 266  (Online  oonnec'bion  'time) 
0.250  (Off linn  cdnnec'bion  'time.') 
0.516  (Total  oonn«?ction  'time) 


G.  Q029008.039 

1A36A63 
1402491 
1366412 
1366093 
1261 300 
1130644 
1069506 

740569 

721339 

720468 

506606 

3835  44 

341824 

308180 

294930 
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VIll.  Conti  I 


Hi     9B29008.DAT  (Includes  only  the  first  part  of  the  file) 

1569212 
1565222 
1529876 
1C2929^ 
IS  521577 
1515392 
1511*82 
IS 09966 
1507612 
1^,94479 

14909M 
148962S 
1476100 
1471570 
1467354 
1459215 
1452918 
1451(344 
1448820 
14^^8745 
1439545 
1437501 
1437429 
1436463 
1434056 
1434055 
1424745 
1412334 
1410704 
1402491 
i:S96964 
39331 9 
1385383 
1383826 
1368412 
1366C393 
1364561 
1362095 
1359704 
1354162  etc. 
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Vlll.  Cont. 


1.  Q029D08,C0D 


K         15692125   K         15652221   K         1 529876 |   K         1529296;  K 
92;   K         1511482;   K         1509966;   K         1507612J   K  1494479; 
489628;   K         14761 CO;   K         1471570;   K  1467354 
K         1459215;   K         1452918;   K         1451044;   K         1448820;  K 
45;   K         1437501!  K         1437429^  K         1436463^  K  1434056; 

424745;   K         1412334;   K         1410704;   K    1402491 

K         1396964;   K         1 393319 |K         1385383;   K         1383826;  K 
93 I   K         1364561;   K         1362095;  K         1359704;   K  1354162; 

335985;   K         1332280;^  K         1305705;   K   1305533 

K         1303785;   K         1303058;   K         1301209;   K         12992701  K 
27;   K         1281300;   K         12'^9025§   K         1277749;   K  1272060; 
262237;   K         1255066;   K         1 25035 -V;   K  1238285 


K         1237376;   K         1 229866 j  K 
08;   K         1169669;   K         1 1601 94 | 
130844?   K         1124464;   K  112S 
K         1101445;   K         1091166;  K 
961  K         101 6245 J   K         10159*?  , 


12294421  K 


7': 


1219135;  K 
1150698;   K  1145539; 
K  1110065 
-'SDt^i  K         1081 393 J-  K 
1010858;   K  967579; 
962458;   K  957783;   K  9E7777|   K  956873 

K  954047;   K  942254?   K  9328  75;   K  914'f'^7;  K 

03;   K  S66642;   K  865602;  K  854915 |   K  851530; 

S4A102;   K  844066;   K  S42903;   K  831822 

K  826940;   K  823348;   K  820577i   K  812631;  K 

92;  »\  788940:   K  7S8787;   K  7836  24;   K  776078; 

7669<?6j   K  766116;   K  766017;   K  763170 

k  7631651  K  760780 i  K  757947;   K  749936;  K 

66 i  k  732985;   K  732947;   K  7307ob;   K  730022; 

720468;   K  714642;   K  707253;   K  696210 

K  674461 jK  667233;   K  665462;   K  649848;  K 

57j   K  633767;   K  629522;   K  595557;   K  59482?:^ 

5876731   K  584196;   K  572041 |   K  570129 

K  566548;   K  566169;   K  565886;   K  562724;  K 

27;   K  552540;   K  545300;   K  542319;   K  535297; 

525213;   K    5252105   K  525195;   K  510401 

K  508497;   K  506806;   K  506147;   K  504488;  K 

57;   k  482630;   k  466838;   K  459415;   K  442755; 

436117;   k  425388;   K  423757;   K  421571 

k  415096;   k  415089;   K  414035;   k  411958;  K 

44;   k  369605;   K  359976;   K  357195;   K  341 S55; 

332024;   k  328011 J   k  31 8584 jK  315213 

k  311964;   k  3081 8C;   K  307377;   K  2998041 


K 


30 ;   K  282^71;   K  279655;   K  233129;   K  218772; 

1984331   k  196528;   k   195842;   K  195558 

k  192980;   k  191778;   K  188497;   K  182393;  K 

17;   k  140453;   K  122998;   K  103264;   K  094715; 

0890411  k  0880481  K _ _       087775;   K  068622 

k  064840;   K  061004 


15215771  K  1! 

K         1490941;  K 

1448745;   K  1' 

K         1434055;  K 


1368412;  K 
K  1349654; 


1; 


K 


1295624;  K 
K  1262238; 

1197294;  K 
K  1136726; 

1039863;  K 
K  964093 ; 

912724;  K 
K  844931 ; 

791110;  k 
K  769671  I 

74C3589J  K 
k  721339; 

645662;  K 
K  592929; 

560634;  k 
K  526419; 

503582;  K 
K  436332; 

410112;  k 
K  341824; 

29788'?;  K 
K  214196; 

166340;  K 
K  094245 ; 


ir 


K 


1^ 


K 


It 


K 


K 


K 


K 


K 


K 


K 


K 


K 
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Vlil.  Cont. 


J.     Fd29dd8 . ABS  (includes  only  the  first  part  of  the  file) 


0/5/1 

1569212       E.I.   Monthly  No:  EI8512120?6S 

MICROWAVE     MEASUREI1ENT     OF  DIF  ;  .     .rC  PROPERTIES  OF  LOW-LOSS  MATERIALS  BY 
THE  BlELECTRie  ROD  RESONATOR  KETi- 

Robayashi,   Yoshib;  Katoh ,  Masayui<:^ 

Saitama  Uriiv,   Dep  of  Electrical  Engineering,  Urawa,  Jph 

£EEE  Trans&:::3tiDnB  on  MicrbvJftve  Theory  and  Techniques  v  MTT-33  n  7  July 
1985  p  586-592 

CODEN:   lETMAB       ISSN:  0018-9480 
benguage:  ENGLISH 

Dodumeht  Type:  JA;    (Journal  Article)       T?eatm'ent:   T;    (Theoretical);   X;    (  — 
Experimental > 

Improvernents  in  both  accuracy  and  speed  are  described  for  the  technique 
of  measuring  the  microwave  dielectric  properties  of  low— loss  materials  by 
using  a  dielectric  rod  resonator  short-circuited  at  both  ends  by  two 
paraliel  conducting  plates.  A  technique  for  measuring  the  effective  surface 
resistance  R//s  of  the  conducting  plates  is  proposed  to  allow  the  accurate 
measurement  of  the  loss  tangent  tan  DELTA  .  By  means  of  a  first  -order 
approximation,  expressions  are  analv  *-.ical  ly  derived  for  estifPrk^^^ing  the 
errors  of  the  measured  vaiues  of  relative  permittivity  FPSILON  //r  ^  tan 
DELTA  ,  and  R//s  ,  for  measuring  the  temperature  coefficient  of  EPSILON  //r 
,  and  for  determining  the  required  size  of  the  conducting  plates. 
Computer-aided  measurements  are  realized  by  using  thete  expressions.  It  is 
shown  that  the  temperature  dependence  of  R//s  should  be  considered  in  tan 
DELTA  measurementsi  Experimental  results  are  given  for  a  99.  9-%  alumina 
ceramic  rod  sample.  12  refs. 

Desci^iptors:  ^MICROWAVE  MEASUREMENTS;  MATERIALS--Dielectric  Properties; 
RESOr^ATORS—Applicabibhs;  MATHEMATICAL  TECHNIQUES—Approximatipn  Theory; 
TEMPERATURE  MEASUREMENT--eomputei    Applications;  MEASUREMENT  ERRORS   

Identifiers:  DIELECTRIC  ROD  RESONATOR;  LOW  .OSS  MATERIALS;  LOSS  TAN<5ENT| 
PARALLEL  CONDUCTING  PLATES  Sh  ^T-CIRCUIT;  ERROR  ESTIMATION  OF  MEASURED 
VALUES 

Classsif ication       Codes:       942         (Electrical       &       Electronic  Measuring 
instruments);      rOl        (Electricity     &     Magnetism);  70S     (Electric  &  Magnetic 
Materials);     921        (Applied     Mathematics);     723       (Computer     Software);  944 
(Moisture,   Pressure  &  Temperature,   &  Radiation  Measuring  Instruments) 
94         (INSTRUMENTS     &     MEASUREMENT);     70       (ELECTRICAL     ENGINEERING);  92 
(ENGINEERING  MATHEMATICS);   72     (COMPUTERS  &  DATA  PROCESSING) 

RELEVANT  PARTIALLY  RELEVANT         _  NOT  RELEVANT^ 
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F029008 - ABS   <  cont - ) 
b/S/2 

156S222       E-I.   Monthly  No:  £18512120261 

MICROWAVE  toss  QUAtiTY  OF  BaZn//i /////3Ta//2/////3d//3  CERAMiCSi 
Desa,  Ai  B. ;   0' Bryan,   Hi  Mi 

AT&T  Technologies,   Engineering  Resear       Cent,   Princeton,   NJ,  USA 
Journal  of  the  American  Ceramic  Society  v  68  n  10  Oct  1985  p  546-551 
eODEN:   JACTAW       iSSN:  0002-7820 
Languages  ENGLISH 

Document  Type :  JA;  (Journal  Article)  Treatment:  A;  (Applications);  X;  - 
(Experimental) 

?D_  "the  _P^st  few  years  a  number  of  materials  have  been  developed  as 
fnicrowav;-:  dielectrics.  For  these  compositions  with  Zn  it  was  found  that  the 
microwave  loss  _goal ity  Q  improved  by  sintering  at  very  bigh 
temperatures  or  f or  1  onger  times  at  1  ower  temperatures •  The  incre^ised  Q  has 
been  explained  in  terms  of  ordering  of  Zn  and  Ta  ions  on  the  B  site  of  the 
peroyskite  lattice.   10  refsi   

bescr i  ptors :  *MI CROWA VE  DEVI CES~*Manu f  acture ;  CERAMI C  MATERIALS — 
Sintering;  OSCILLATORS,   MICOWAVE — l=>erf ormance 

Identifiers:  CERAMIC  RESONATORS;  SINTERIN6  PROCESS  CONTROL;  PEROVSKITE 
t^^iTtiCE  

V.  i^.ssif  ication  Codes:       71 -h^       (Eiec 'ironic     ^P^PPP^D'^^'^  5  ^"^^ 

'*;?^ntronic     Equipment);     812       (Ceramics     &  Refractories)   713  (Electronic 
C-ivCUits) 

71      (ELECTRONICS  &   tJMMUNif:^  "IONS)  ;    81      (CHEMICAL   PROCESS   INDUST*;  ^ 
RELEVANT  PARTIALLY  RELEVANT  MOT  RELEVANT 
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VIII.  Cont. 


k.  F029d0S.039 

1436A63P  14b2A«?lR  13684121x1  1366093P  1281300R  1136844R 
108950aP 

7Ab589N         721339R         72d468R         506806E         3835 AAE  3A182AE 
3081 80E  29A930E 


L .     F029008 . DAT 


156921 2R 
1511482R 
lA76lbbR 
14A8820P 
j434056N 
139696Aivi 
1364561 P 
1332280N 
1299270N 
1272060P 
1237376R 
1169669N 
1124A6AR 
108i393P 
967579P 
954b47R 
S66642N 
844b66R 
31 2681 N 
776b78P 
7631 65N 
732985 R 
71 4642 P 
64934SP 
594821 P 
566548E 
B52S4bE- 
52521 OE 
5b4488E 
44275SE 
41 Sb96E 
3696b5E 
328B11E 
99804E 
i-i8772e 
19298dE 
140453E 
Q88648E 


15652:??P 

1509V66jsl 

1471 5 70N 

144874SN 

1434055N 

1 39331 9R 

1 362095 R 

130570t>ivl 

1295 624 R 

1262238N 

1229866P 

i 1601 94 P 

1123756N 

i 039863 P 

964093 P 

942254N 

8656b2N 

S429b3N 

791 11 ON 

769671 P 

76b78bN 

73294 7N 

7b7253R 

645662P 

5 92929 R 

5661 69E 

545300E 

5251 95 E 

503582E 

436332E 

415039E 

i?S9976E 

316584E 

297889E 

214196E 

i9^778E 

122998E 

b87775E 


15 29876 R 

15b7612N 

1467354R 

1439545N 

1424745N 

1385383N 

13597b4R 

1305533P 

1285727P 

1262237N 

1 229442 P 

1150698P 

■Ml  0065 N 

lb31696N 

96245 8N 

932875N 

85491 5N 

831 822N 

791 092 R 

766996R 

7S7947P 

730766N 

69621 ON 

639957R 

587673N 

565886E 

54231 9E 

510401 E 

492857E 

4361 17E 

414035E 

3571 95E 

31521 3E 

294930E 

19S433E 

188497E 

103264E 

b68622E 


1529296N 
1494479P 

1 45 921 5 N 

1 437501 R 

1412334N 

1363826P 

1354162N 

130Z;785P 

1281300R 

1 25 5 066 R 

1 21 91 35  R 

1145539P 

1 101 445 N 

1016245N 

95 7^62 R 

9.i4497N 

851530N 

826940N 

788940N 

7661 16N 

749936N 

730022R 

674461 N 

633767N 

5841 96N 

562724E 

53E297E 

503497E 

4e2630E 

425388E 

411958E 

341S55E 

311964E 

282571 E 

196528E 

•i82393E 

09471 5 E 

Q6484QE 


1521F7.'N 
1 490941 N 
1452918P 
1437429R 
14ig704N 
1 36841 2N 
1349654N 

isqsossp 

1 279025 R 

1250334N 

1197294N 

113672t>N 

1091166IM 

1015991 P 

957777N 

912724N 

844931 N 

823348N 

7S8787P 

76601 7N 

740589N 

721 339 R 

667233N 

629522R 

572041 N 

560634E 

52641 9E 

506806E 

466838E 

423757E 

410112E 

341S24E 

3081 80E 

27965 5 E 

195842E 

i6634bE 

094245 E 

b6lbb4E 


1515392R 

148 9628 P 

1451044P 

1436463P 

1 402491 R 

1366g93P 

133B9C5N 

1301209N 

1277749N 

1238285R 

1175008N 

113gS44R 

1089B08P 

1010S58N 

9156873R 

8840b3N 

8441 02N 

820577N 

783824R 

7631 70N 

734406N 

720468f^ 

665462N 

595557N 

5701 29N 

556527E 

525213E 

5061 47E 

45941 5 E 

421 571 E 

383544E 

r5320  24E 

3b7377E 

2331  29E: 

195558E 

15071 7E 

089041 E 


VIII*  Conti 


ti.  QD29e08iOUT 

1569212  RELEVANT 

1565222  PSRTIALLY  RELEVANT 

1529876  RELEVANT 

1529296  NOT  RELEVANT 

1521577  HOT  RELEVANT 

1515392  RELEVANT 

15115S2  RELEVANT 

1509966  NOT  RELEVANT 

1507612  NOT  &ZLEVANT 

149^479  PARTIALLY  RELEVANT 

l«Sii7?l  NOT  RELEVANT 

1^89628  PARTIALLY  RELEVANT 

1-^76100  RELEVANT 

1-^71570  NOT  RELEVANT 

1-^67355  RELEVANT 

l«92i5  ISOT  RELEVANT 

l«2?ie  PARTIALLY  RELEVANT 

14510^4  PARTIALLY  RELEVANT 

-^8320  PARTIALLY  RELEVANT 

587*3  NOT  RELEVANT 
1439545   ■      NOT  RELE^'ANT 

1437501  RELEVANT 

1437429  RELEVANT 

1436463  PARTIALLY  RELEVANT 

1434056  NOT  RELEVANT 

1434055  NOI  RELEVANT 

1424745  NOT  RELEVANT 

1412334  NOT  RELEVANT 

1410704  NOT  RELEVANT 

1402491  RELEVANT 

1396964  NOT  RELEVANT 

139331?  RELEVANT 

1385383  NOT  RELEVANT 

1383826  PARTIALLY  RELEVANT 

1368412  NOT  RELEVANT 

1366093  PARTIALLY  RELEVANT 

1364561  PARTIALLY  RELEVANT 

1362095  RELEVANT 

1359704  RELEVANT 

1354162  NOT  RELEVANT 

1349654  NOT  RELEVANT 

1335985  NOT  RELEVANT 

1332280  NOT  RELEVANT 


Procedure  12  -  p.  42 


VI 11.  Conti 


i}D290DS.0UT  Ccoht.S 


13057D5  NOT  RELw.VANT 

1305533  PABTIAL'  Y  RELEVANT 

1303785  PARTIALLY  RELEVANT 

1303058  PARTIALLY  RELEVANT 

1301209  NOT  RELEVANT 

1299270  NOT  RELEVANT 

1295625  RELEVANT 

1285727  PARTIALLY  RELEVANT 

1281300  RELEVANT 

1279025  RELEVANT 

1277759  NOT  RE:LEVANT 

1272060  PARTIALLY  RELEVANT 

1262236  NOT  RELEVANT 

1262237  NOT  RELEVANT 
1255066  RELEVANT 
1250335  NOT  RELEVANT 
1238285  RELEVANT 
1237376  RELEVANT 
1229866  PARTIALLY  RELEVANT 
1229552  PARTIALLY  RELEVANT 
1219135  RELEVANT 
1197295  NOT  RELEVANT 
1175008  NOT  RELF«  '5»NT 
1169669  NOT  RELEV  ".IV 
1160195  PARTIALLY  :*ci  EVANT 
1150698  PARTIALLY  RELEVANT 
1155539  PARTIALLY  RELEVANT 
1136726  NOT  RELEVANT 
1130855  RELEVANT 
1125565  RELEVANT 
1123756  NOT  RELEVANT 
1110065  NOT  RELEVANT 
1101555  NOT  RELEVANT 
1091166  NOT  RELEVANT 
10S95Q8  PARTIALLY  RELEVANT 
1081393  PARTIALLY  RELEVANT 
1039863  PARTIALLY  RELEVANT 
1031696  NOT  RELEVANT 
1016255  NOT  RELEVANT 
1015991  PARTIALLY  RELEVANT 
101Q858  WOT  RELEVANT 

967579  PARTIALLY  RELEV?.NT 

965093  PARTIALLY  RELEVANT 
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VIII.  Cont. 


Qd29008.dtlT  <cont.? 


962458  NOT  REbEVANT 

957783  REtEVANT 

957777  NOT  REtEVANT 

956873  RELEVANT 

954S47  RELEVANT 

942254  NOT  RELEVANT 

932875  NOT  RELEVANT 

914497  NOT  RELEVANT 

912724  NOT  RELEVANT 

884QB3  NOT  RELEVANT 

866642  NOT  RELEVANT 

865602  MOT  RELEVANT 

85491 S  NOT  RELEVANT 

851530  NOT  RELEVANT 

844931  NOT  RELEVANT 

844102  NOT  RELEVANT 

844066  RELEVANT 

S42903  NOT  RELEVANT 

831822  NOT  RELEVANT 

826940  NOT  RELEVANT 

823348  MOT  RELEVANT 

820577  NOT  RELEVANT 

81 2681  NOT  RELEVANT 

791110  NOT  RELEVANT 

791092  RELEVANT 

788940  NOT  RELEVANT  

768787  PARTIALLY  RELEVANT 

783824  RELEVANT  

776078  PARTIALLY  RELEVANT 

769671  PARTIALLY  RELEVANT 

766996  RELEVANT 

766116  NJT  RELEVANT 

766017  NOT  RELEVANT 

763170  NOT  RELEVANT 

763165  NOT  RELEVANT 

760780  NOT  RELEVANT 

757947  PARTIALLY  r?ELEVANT 

749936  NOT  RELEVANT 

740589  NOT  RELEVANT 

734466  NOT  RELEVANT 

7329SE  RELEVANT 

732947  5>iOT  RELEVANT 
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VIII.  Conti 
Qd29dd8.6UT  <cont.) 


NOT 

RELEVANT 

V30S22 

RELEVANT 

.2-4339 

REtEVANT 

RELEVANT 

714642 

PARTiALLY  RELEVANT 

707253 

RELEVANT 

696210 

NOT 

RELEVANT 

674461 

NOT 

RELEVANT 

667233 

NOT 

RELEVANT 

665462 

NOT 

RELEVANT 

649848 

PARTIALLY  RELEVANT 

645662 

PARTIALLY  RELEVANT 

639957 

RELEVANT 

633767 

NOT 

RELEVANT 

629522 

RELEVANT 

595557 

NOT 

RELEVANT 

594821 

PARTIALLY  RELEVANT 

592929 

RELEVANT 

587673 

NOT 

RELEVANT 

584196 

NOT 

RELEVANT 

572041 

NOT 

RELEVANT 

570129 

NOT 

RELEVANT 

566548 

NOT 

EVALUATED 

566169 

NOT 

EVALUATED 

565886 

NOT 

EVALUATED 

562724 

NOT 

EVALUATED 

560634 

NOT 

EVALUATED 

556527 

NOT 

EVALUATED 

552540 

NOT 

EVALUATED 

545300 

NOT 

EVALUATED 

542319 

NOT 

EVALUATED 

535297 

NOT 

EVALUATED 

526419 

NOT 

EVALUATED 

525213 

NOT 

EVALUATED 

525210 

NOT 

EVALUATED 

525195 

NOT 

EVALUATED 

510401 

NOT 

EVALUATED 

508497 

NOT 

EVALUATED 

506806 

NOT 

EVALUATED 

506147 

NOT 

EVALUATED 

504488 

NOT 

EVALUATED 

50353^: 

NOT 

EVALUATED 

492857 

NOT 

EVALUATED 

482630 

NOT 

EVALUATED 

ERIC 


VIII.  Cont; 


Procedure  12  -  p.  A5 


Q029008.0UT  <cont.) 


466838 

NOT 

EVALUATED 

4S941S 

MOT 

EVALUATED 

4427BR 

MOT 

EVALUATED 

NOT 

EVALUATED 

NOT 

EVALUATED 

NOT 

EVALUATED 

4237S7 

NOT 

EVALUATED 

421 B71 

NOT 

EVALUATED 

41 B 096 

NOT 

EVALUATED 

41BbS9 

NOT 

EVALUATED 

41403B 

NOT 

EVALUATED 

4119S8 

NOT 

EVALUATED 

410112 

NOT 

EVALUATED 

383544 

NOT 

EVALUATED 

369605 

NOT 

EVALUATED 

359976 

NOT 

EVALUATED 

3571 95 

NOT 

EVALUATED 

341855 

NOT 

EVALUATED 

341824 

NOT 

EVALUATED 

332024 

NOT 

EVALUATED 

328011 

NOT 

EVALUATED 

318584 

NOT 

EVALUATED 

315213 

ivioT 

EVALUATED 

311964 

NOT 

EVALUATED 

308180 

NOT 

EVALUATED 

30/377 

NOT 

EVALUATED 

299804 

NOT 

EVALUATED 

297889 

NOT 

EVALUATED 

f:?4930 

NOT 

EVALUATED 

282571 

NOT 

EVALUATED 

279655 

NOT 

EVAtUATED 

233129 

NOT 

EVALUATED 

218772 

NOT 

EVALUATED 

214196 

NOT 

EVALUATED 

198433 

NOT 

EVALUATED 

196528 

NOT 

EVALUATED 

195842 

NOT 

EVALUATED 

195553 

NOT 

EVALUATED 

19298'3 

NOT 

EVALUATED 

191778 

NOT 

EVALUATED 

188497 

NOT 

EVALUATED 

182393 

NOT 

EVALUATED 

1 66340 

NOT 

EVALUATED 
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Procedure  12  -p.  46 


Q029006.0UT  Cobht.7 


AZ>%JfA  f 

POAI  1  l&TPn 
ELVrlLUri  1  d.  L.' 

140453 

NOT 

EVALUATED 

122998 

NOT 

EVALUATED 

103264 

NOT 

EVALUATED 

094715 

NOT 

EVALUATED 

094245 

NOT 

EVALUATED 

089041 

NOT 

EVALUATED 

088048 

NOT 

EVALUATED 

087775 

NOT 

EVALUATED 

068622 

NOT 

EVALUATED 

064840 

NOT 

EVALUATED 

061004 

NOT 

EVALUATED 

N.  D029008.039 

rnicrowaves 

ceramic  ma'teriais 

ceramic  ma-teria Is  -  sintering 

ceramic? 

f  irt 

f  iring_ 

sinter? 

microwave? 

radiation 
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Proce<'.ure  12  -  p.  47 


O.     SEARD39i0tlT  (Includes  only  two  questions) 

Searcher  No.  039 


QUEST-DBASE 
NUriBER 

022108 


029008 


RELV 
IS 


36 


m        TOTftL   #        TOTAL  U 

TOTAL  # 

PREL  NREL 

EVAL 

135  0 

150 

#  of   rel evan't  itpems 

b 

#  of  oartial ly  relv 

6 

^     c       not  relevant 

■ 
• 

d 

tot a 1  #  eva 1 uated 

■ 

m 

6 

#     not  evaluated 

m 
m 

64 

total   #  retrieved 

s 

70 

recal 1 

■ 

m 

0.040 

precision 

m 
m 

1  .000 

total  #  of  search  terms 

9 

tbta    #  of  commands 

m 
m 

26 

total  4  of  cycles 

u 

m 

6 

Dn""lin©  connect  time 

■ 

0^306 

of f  prepara wion  time 

■ 
■ 

0 . 250 

total   ^earcHino  time 

0.556 

34  so 

150 

#  of   rel evant  items 

5 

#  of  partially  relv 

■ 

3 

#     of     not  relevant 

■ 

2 

total  #  evaluated 

■ 

10 

#     not  evaluated 

5 

total   #  retrieved 

■ 

15 

recal 1 

■ 
■ 

0  i  i  i  4 

precision 

■ 
■ 

0.800 

total  #  of  search  terms 

■ 

9 

total  #  of  commands 

■ 

25 

tot a 1  #  of  eye 1 es 

s 

6 

on-line  connect  time 

■ 

0.268 

of  f  preparation  time 

■ 
• 

0 . 250 

tota 1  search ing  time 

■ 

m 

0.518 

TOTAL  « 
NEVL 

365 


TOTAL  m 
RETRIEVEC 

515 


77 


227 


ERIC 


576 


Procedure  12  -  p.  48 


Vlil.  Cont. 


P.  QUES029.DUT 

Quesibibh  No.  02? 
Dat^abase  No.  OOB 


*  SUfhfhary  of  Search  Re5Ult.5  * 
******************************* 


Total 


Relevant,  abst.ract.s 

i  36 

Partially  relevant 

:  34 

# 

Not  relevant 

:  60 

Evaluated 

:  150 

Not  evaluated 

s  77 

of  references 

:  227 

User  evalua-tion: 


User^s  time  :  0.75  hrs. 

bollar  Value  assigned  :  $200.00 

Worth  assigned  s  5 

Problem  resolution  :  4 

Satisfaction  :  4 


Searcher  evaluation: 


As  #  Relevant 

Bz  #  Partially  relev^ 

C:  #  Not  relevant 

D:  Total  #  evaluated 

E:  #  Not  evaluated 

Fz  Total  #  retrieved 

G :  Reca 1 1 


H:  Precision  

I:  Total  #  commands 

J:  Total  #  cycles 

K :  Tota 1  #  search  terms 

L :  On 1 i ne  connect  time 

Ms  Preparation  time 

Ns  Total  time 


SEAR 

A 

B 

C 

D 

E 

F 

G 

H 

i 

j 

k 

t 

M 

N 

Oil 

8 

8 

12 

28 

22 

50 

0.229 

0.571 

32 

2 

24 

0.076 

0.083 

0.159 

027 

15 

13 

50 

78 

23 

101 

0.400 

0.359 

19 

7 

15 

0.161 

0.250 

6.411 

038 

3 

1 

1 

5 

5 

8 

0.057 

0.800 

17 

6 

9 

0.326 

0.167 

6.493 

039 

5 

3 

2 

IG 

5 

15 

0.114 

0.800 

25 

6 

9 

0.268 

0.250 

0.518 

o?>o 

3 

0 

0 

3 

0 

3 

0.042 

1 .000 

30 

9 

22 

0.423 

0.333 

0.756 

120 

28 

10 

13 

51 

24 

75 

0.543 

b .  745 

9 

4 

5 

0.198 

0.25O 

0.448 

220 

7 

4 

1 

12 

5 

17 

0.157 

0.917 

7 

2 

13 

0.1S6 

0.250 

0.406 

320 

19 

9 

15 

43 

33 

76 

0.400 

0.651 

10 

1 

17 

0.258 

0.283 

0.541 

420 

6 

5 

2 

13 

0 

13 

0.157 

0.846 

11 

17 

0.168 

0.417 

6.605 

577 


